Nov 03 20:26:32.231233 ip-10-0-128-118 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 20:26:32.231246 ip-10-0-128-118 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 20:26:32.231255 ip-10-0-128-118 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 20:26:32.231579 ip-10-0-128-118 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 20:26:42.348928 ip-10-0-128-118 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 20:26:42.348946 ip-10-0-128-118 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8150a4b3d76a4ec793561c3a9092f457 --
Nov 03 20:28:51.949346 ip-10-0-128-118 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:28:52.239360 ip-10-0-128-118 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:52.239360 ip-10-0-128-118 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:28:52.239360 ip-10-0-128-118 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:52.239360 ip-10-0-128-118 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:52.243759 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243680    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.243759 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243756    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.243759 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243760    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.243759 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243764    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243768    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243771    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243773    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243776    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243779    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243781    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243784    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243786    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243790    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243792    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243795    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243797    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243800    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243802    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243805    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243807    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243810    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243813    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243815    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.243931 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243818    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243829    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243831    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243834    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243836    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243839    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243842    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243844    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243847    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243849    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243852    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243855    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243858    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243861    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243863    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243866    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243869    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243871    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243874    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243877    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.244413 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243879    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243883    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243887    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243890    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243893    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243896    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243899    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243902    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243904    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243907    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243910    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243912    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243914    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243917    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243920    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243922    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243927    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243930    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243934    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.244876 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243937    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243939    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243942    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243945    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243947    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243950    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243953    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243956    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243960    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243962    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243965    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243967    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243970    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243972    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243975    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243978    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243980    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243983    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243985    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.245347 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243987    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243990    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243993    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243996    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.243999    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244001    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244004    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244007    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244010    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244012    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244015    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244017    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244020    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244023    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244026    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244028    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244031    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244047    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244052    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.245851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244057    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244061    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244066    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244070    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244074    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244079    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244836    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244843    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244846    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244850    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244852    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244855    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244858    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244861    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244863    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244866    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244869    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244872    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244874    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244877    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.246427 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244879    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244882    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244885    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244887    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244890    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244892    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244894    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244897    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244900    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244902    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244905    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244908    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244910    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244914    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244917    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244919    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244922    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244924    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244927    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244930    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.246927 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244932    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244935    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244937    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244939    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244942    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244944    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244947    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244949    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244952    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244955    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244957    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244960    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244962    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244965    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244967    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244970    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244972    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244975    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244977    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244980    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.247433 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244982    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244985    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244987    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244992    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244994    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244997    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.244999    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245002    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245007    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245010    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245013    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245016    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245019    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245021    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245024    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245026    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245029    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245049    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245053    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.247911 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245058    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245061    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245063    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245067    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245070    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245073    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245075    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245078    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245080    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245083    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245085    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245088    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245090    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245092    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245095    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245098    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245101    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245103    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245106    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245109    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.248380 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245112    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245114    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245117    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245119    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245122    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245124    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245127    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245129    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245132    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245136    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245140    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245143    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245146    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245617    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245622    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245626    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245629    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245631    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.248851 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245634    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245636    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245639    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245641    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245645    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245647    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245650    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245652    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245655    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245657    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245661    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245663    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245666    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245669    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245672    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245674    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245677    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245679    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245682    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.249299 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245684    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245687    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245689    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245692    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245694    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245697    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245699    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245702    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245704    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245707    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245710    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245713    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245715    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245718    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245720    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245722    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245725    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245728    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245732    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.249750 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245736    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245739    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245741    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245744    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245746    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245749    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245752    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245755    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245757    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245760    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245762    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245765    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245768    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245770    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245772    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245775    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245778    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245780    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245783    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245785    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.250214 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245789    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245791    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245794    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245796    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245798    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245801    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245804    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245806    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245809    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245811    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245813    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245816    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245819    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245821    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245824    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245827    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245830    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245834    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245837    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.250782 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245839    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245842    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245844    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245847    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245849    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245852    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245855    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245857    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245860    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245862    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245865    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245868    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245871    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245874    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245876    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245879    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245882    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245885    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245887    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.251287 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245890    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245893    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245896    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245898    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.245900    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247120    2567 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247131    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247139    2567 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247144    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247148    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247151    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247155    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247160    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247163    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247166    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247169    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247173    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247176    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247179    2567 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247181    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247185    2567 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247188    2567 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247191    2567 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247194    2567 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:28:52.251725 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247198    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247201    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247204    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247208    2567 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247212    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247215    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247218    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247221    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247224    2567 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247228    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247231    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247234    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247237    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247241    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247244    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247247    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247250    2567 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247253    2567 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247257    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247261    2567 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247264    2567 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247267    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247270    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247273    2567 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:28:52.252298 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247278    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247281    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247284    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247287    2567 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247290    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247293    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247296    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247299    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247302    2567 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247305    2567 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247308    2567 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247315    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247318    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247321    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247324    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247327    2567 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247330    2567 flags.go:64] FLAG: --help="false"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247333    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247337    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247340    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247343    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247347    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247350    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:28:52.252890 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247353    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247356    2567 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247359    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247362    2567 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247365    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247368    2567 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247371    2567 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247374    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247377    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247382    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247385    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247388    2567 flags.go:64] FLAG: --lock-file=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247390    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247393    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247396    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247402    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247404    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247407    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247410    2567 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247413    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247416    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247419    2567 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247422    2567 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247427    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247430    2567 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:28:52.253455 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247433    2567 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247436    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247440    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247444    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247448    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247451    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247454    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247457    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247464    2567 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247467    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247470    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247473    2567 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247476    2567 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247479    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247482    2567 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247485    2567 flags.go:64] FLAG: --port="10250"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247489    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247492    2567 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05edf57cb66b72027"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247495    2567 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247498    2567 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247501    2567 flags.go:64] FLAG: --register-node="true"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247504    2567 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247508    2567 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247511    2567 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247514    2567 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247516    2567 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:28:52.254044 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247520    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247523    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247526    2567 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247529    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247532    2567 flags.go:64] FLAG: --runonce="false"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247535    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247538    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247541    2567 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247544    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247549    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247553    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247556    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247559    2567 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247562    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247565    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247568    2567 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247571    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247574    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247577    2567 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247580    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247585    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247588    2567 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247593    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247596    2567 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247599    2567 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:28:52.254695 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247602    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247605    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247608    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247611    2567 flags.go:64] FLAG: --v="2"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247615    2567 flags.go:64] FLAG: --version="false"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247620    2567 flags.go:64] FLAG: --vmodule=""
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247624    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.247627    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247764    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247768    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247772    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247775    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247778    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247780    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247783    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247785    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247790    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247792    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247795    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247798    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247801    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247803    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.255311 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247806    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247808    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247810    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247813    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247816    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247818    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247821    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247825    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247828    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247830    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247833    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247835    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247838    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247840    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247843    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247846    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247848    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247851    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247853    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.255833 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247856    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247859    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247861    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247864    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247866    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247870    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247874    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247878    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247881    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247884    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247887    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247890    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247892    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247895    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247898    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247900    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247903    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247905    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247908    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.256292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247911    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247914    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247917    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247920    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247923    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247925    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247928    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247931    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247933    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247936    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247938    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247941    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247943    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247946    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247950    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247953    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247956    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247959    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247961    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247964    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.256737 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247968    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247970    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247973    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247976    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247979    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247981    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247984    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247986    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247989    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247992    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247994    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.247997    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248000    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248003    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248006    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248008    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248011    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248014    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248016    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.257226 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248019    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248021    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248024    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248026    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248029    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248043    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248046    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248048    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248051    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248054    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248056    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248059    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248061    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248066    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248068    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.257660 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.248095    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248133    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248139    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248145    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248149    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248151    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248154    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248157    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248159    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248162    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248165    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248169    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248172    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248174    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248177    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248180    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248182    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248185    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248187    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.258017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248190    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248193    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248195    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248198    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248200    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248203    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248205    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248208    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248211    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248213    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248217    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248220    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248225    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248228    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248230    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248233    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248236    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248239    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248241    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.258486 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248244    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248246    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248249    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248251    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248254    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248258    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248261    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248263    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248266    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248269    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248271    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248274    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248276    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248278    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248282    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248284    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248287    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248290    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248292    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.258939 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248295    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248297    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248300    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248302    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248305    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248307    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248311    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248313    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248316    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248318    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248321    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248323    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248326    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248329    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248332    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248334    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248337    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248339    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248343    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248346    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.259408 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248348    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248351    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248353    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248356    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248359    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248361    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248364    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248367    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248369    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248372    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248374    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248377    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248380    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248382    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248385    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248387    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248390    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248393    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248396    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.259905 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248399    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248402    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248404    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248407    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248410    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248412    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248415    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248418    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248421    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248423    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.248425    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.254669    2567 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.254685    2567 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.255278    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.255284    2567 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255343    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255348    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255352    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255356    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.260350 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255359    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255362    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255365    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255368    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255371    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255374    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255377    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255379    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255382    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255385    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255387    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255390    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255393    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255395    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255398    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255400    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255403    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255405    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255408    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.260811 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255410    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255413    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255416    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255418    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255421    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255423    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255432    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255435    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255438    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255441    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255443    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255446    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255449    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255451    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255454    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255457    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255459    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255462    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255464    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.261291 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255467    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255470    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255472    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255474    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255477    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255480    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255483    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255485    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255488    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255490    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255493    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255495    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255498    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255500    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255503    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255505    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255508    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255510    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255513    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255516    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.261732 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255524    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255527    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255530    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255532    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255535    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255537    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255540    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255542    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255545    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255548    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255550    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255553    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255555    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255558    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255560    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255563    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255567    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255571    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255574    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255576    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.262223 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255579    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255582    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255584    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255587    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255589    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255592    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255596    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255600    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255603    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255606    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255609    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255612    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255615    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255623    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255625    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255628    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255630    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255633    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255635    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255638    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.262823 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255640    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255643    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255645    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255648    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.255677    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255850    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255855    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255858    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255860    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255863    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255865    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255868    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255871    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255873    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255876    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255879    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.263614 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255881    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255884    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255886    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255889    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255891    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255894    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255896    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255899    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255902    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255904    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255912    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255915    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255917    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255920    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255922    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255925    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255927    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255930    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255933    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255935    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.264195 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255938    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255940    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255942    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255945    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255947    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255950    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255952    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255955    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255957    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255960    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255963    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255967    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255970    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255972    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255975    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255977    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255980    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255982    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255985    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255987    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.264660 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255990    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255992    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.255995    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256002    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256005    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256008    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256011    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256013    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256015    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256018    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256020    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256023    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256025    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256028    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256030    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256044    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256048    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256052    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256055    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.265292 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256058    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256061    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256063    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256065    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256068    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256071    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256079    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256082    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256085    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256088    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256090    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256093    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256095    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256098    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256100    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256103    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256105    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256115    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256117    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256120    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.265767 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256122    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256125    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256127    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256130    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256132    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256135    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256137    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256140    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256142    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256145    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256147    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256150    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256153    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256155    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256158    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.266261 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256160    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.256184    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256240    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256244    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256248    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256250    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256254    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256256    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256259    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256261    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256264    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256267    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256270    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256272    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256275    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256284    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:52.266624 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256287    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256290    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256292    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256295    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256297    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256300    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256303    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256306    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256308    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256311    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256313    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256316    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256318    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256321    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256323    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256326    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256328    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256331    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256333    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:52.267000 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256336    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256339    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256341    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256344    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256346    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256348    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256351    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256354    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256357    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256359    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256362    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256364    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256367    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256375    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256378    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256380    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256383    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256385    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256388    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256390    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:52.267522 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256393    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256395    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256397    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256400    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256402    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256405    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256407    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256410    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256413    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256415    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256417    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256420    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256422    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256425    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256427    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256429    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256434    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256437    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256439    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256442    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:52.267985 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256444    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256448    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256452    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256454    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256457    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256459    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256467    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256470    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256473    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256475    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256478    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256480    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256483    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256486    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256488    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256490    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256493    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256496    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256498    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256501    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:52.268516 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256503    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256506    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256508    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256511    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256515    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256517    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256520    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256524    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256529    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256532    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256535    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256537    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:52.256540    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.256586    2567 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.258486    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.259153    2567 server.go:1014] "Starting client certificate rotation"
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.259252    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:52.269201 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.259757    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:52.269629 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.261120    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:52.269629 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.265649    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:52.273737 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.273725    2567 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:28:52.277143 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.277129    2567 log.go:25] "Validated CRI v1 image API"
Nov 03 20:28:52.277971 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.277950    2567 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:28:52.283196 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.283173    2567 fs.go:135] Filesystem UUIDs: map[3751f397-fdf1-476d-aaaa-0bf8d0afb561:/dev/nvme0n1p4 5585f9de-dc3d-41c8-b742-97bb23eb3e98:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 20:28:52.283263 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.283196    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:28:52.287963 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.287866    2567 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:52.286793296 +0000 UTC m=+0.249875284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095112 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d6b64f3c194930a49e6efe8f8165 SystemUUID:ec27d6b6-4f3c-1949-30a4-9e6efe8f8165 BootID:8150a4b3-d76a-4ec7-9356-1c3a9092f457 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cb:35:82:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cb:35:82:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:34:8b:77:46:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:28:52.287963 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.287962    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:28:52.288088 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.288075    2567 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:28:52.290051 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290015    2567 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:28:52.290190 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290052    2567 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-118.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:28:52.290230 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290192    2567 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:28:52.290230 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290201    2567 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:28:52.290230 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290229    2567 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:52.290306 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290241    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:52.290954 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.290944    2567 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:28:52.294471 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.294459    2567 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:28:52.294583 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.294574    2567 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:28:52.295119 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295109    2567 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:28:52.295163 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295138    2567 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:28:52.295163 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295151    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:28:52.295163 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295160    2567 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:28:52.295290 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295169    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:28:52.295635 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.295622    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:52.297258 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.297239    2567 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:28:52.301915 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.301902    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:52.302522 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.302509    2567 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:28:52.302563 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.302540    2567 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:28:52.303223 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303211    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303231    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303237    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303242    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303249    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303254    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:28:52.303259 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303261    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:28:52.303408 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303267    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:28:52.303408 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303274    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:28:52.303408 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303279    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:28:52.303408 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303287    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:28:52.303408 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303295    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:28:52.303735 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303726    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:28:52.303770 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.303737    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:28:52.307299 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.307287    2567 server.go:1263] "Started kubelet"
Nov 03 20:28:52.307433 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.307399    2567 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:28:52.307601 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.307495    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:28:52.308944 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.308910    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:28:52.309294 ip-10-0-128-118 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:28:52.310460 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.310443    2567 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:28:52.312013 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.311623    2567 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:28:52.316329 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.316314    2567 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:28:52.318973 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.318955    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:52.319336 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.319314    2567 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:28:52.319438 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.319421    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:52.319853 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.319837    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:28:52.319930 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.319889    2567 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:28:52.319930 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.319904    2567 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:28:52.320056 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.319993    2567 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:28:52.320056 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320003    2567 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:28:52.320154 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320118    2567 factory.go:55] Registering systemd factory
Nov 03 20:28:52.320154 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320132    2567 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:28:52.320340 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320323    2567 factory.go:153] Registering CRI-O factory
Nov 03 20:28:52.320340 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320338    2567 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:28:52.320433 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320422    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:28:52.320476 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320441    2567 factory.go:103] Registering Raw factory
Nov 03 20:28:52.320476 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.320460    2567 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:28:52.321274 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.321260    2567 manager.go:319] Starting recovery of all containers
Nov 03 20:28:52.331916 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.331900    2567 manager.go:324] Recovery completed
Nov 03 20:28:52.335928 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.335916    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.337529 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.337501    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.337586 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.337539    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.337586 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.337550    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.338017 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.338004    2567 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:28:52.338066 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.338018    2567 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:28:52.338066 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.338050    2567 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:28:52.338131 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.338072    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.339988 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.339971    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.340080 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.339997    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.340080 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.340007    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.340225 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.340212    2567 policy_none.go:50] "Start"
Nov 03 20:28:52.340270 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.340230    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:28:52.340270 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.340243    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:28:52.342299 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.342286    2567 policy_none.go:44] "Start"
Nov 03 20:28:52.388580 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.388539    2567 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:28:52.392646 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.392633    2567 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:28:52.392720 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.392668    2567 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:28:52.392720 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.392679    2567 server.go:87] "Starting device plugin registration server"
Nov 03 20:28:52.392908 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.392896    2567 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:28:52.392955 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.392910    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:28:52.393053 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.393021    2567 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:28:52.393118 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.393106    2567 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:28:52.393118 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.393118    2567 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:28:52.393517 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.393493    2567 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:28:52.393604 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.393547    2567 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:52.446121 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.446099    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-118.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:28:52.446914 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.446103    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-118.us-west-2.compute.internal.18e4a2f6c2639639  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-118.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-118.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:52.307260985 +0000 UTC m=+0.270342972,LastTimestamp:2026-11-03 20:28:52.307260985 +0000 UTC m=+0.270342972,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-118.us-west-2.compute.internal,}"
Nov 03 20:28:52.491446 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.491399    2567 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-118.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:28:52.493600 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.493587    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.494489 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.494474    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.494562 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.494504    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.494562 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.494543    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.494624 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.494575    2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.527701 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.527678    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:28:52.528843 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.528825    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:28:52.528843 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.528843    2567 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:28:52.528963 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.528860    2567 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:28:52.528963 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.528894    2567 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 20:28:52.533209 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.533124    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-118.us-west-2.compute.internal.18e4a2f6c431686c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-118.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-118.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-118.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:52.337526892 +0000 UTC m=+0.300608879,LastTimestamp:2026-11-03 20:28:52.337526892 +0000 UTC m=+0.300608879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-118.us-west-2.compute.internal,}"
Nov 03 20:28:52.533411 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.533395    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-118.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 20:28:52.574978 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.574962    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:52.575085 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.575056    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kc9pz"
Nov 03 20:28:52.594469 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.594401    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-118.us-west-2.compute.internal.18e4a2f6c431ab27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-118.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-118.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-118.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:52.337543975 +0000 UTC m=+0.300625962,LastTimestamp:2026-11-03 20:28:52.337543975 +0000 UTC m=+0.300625962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-118.us-west-2.compute.internal,}"
Nov 03 20:28:52.594572 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.594448    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 20:28:52.594684 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.594666    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 20:28:52.629628 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.629605    2567 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"]
Nov 03 20:28:52.629702 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.629658    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.630514 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.630497    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.630577 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.630529    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.630577 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.630541    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.632707 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.632696    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.632854 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.632839    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.632899 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.632867    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.633436 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633421    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.633530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633425    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.633530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633472    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.633530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633487    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.633530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633448    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.633530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.633531    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.635559 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.635542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.635636 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.635571    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.636267 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.636250    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.636321 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.636286    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.636321 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.636310    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.668439 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.668422    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.670876 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.670860    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.686565 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.686546    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:52.687144 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.687129    2567 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.721706 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.721687    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.721793 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.721711    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.721793 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.721727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.790331 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.790271    2567 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822058 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822191 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822191 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822104    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822191 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822144    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822310 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822191    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.822310 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.822148    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.888056 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.888012    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:52.888984 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.888967    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:52.889094 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.888995    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:52.889094 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.889008    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:52.889094 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.889048    2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.947515 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.947493    2567 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.947580 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:52.947522    2567 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-118.us-west-2.compute.internal\": node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:52.971518 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.971501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:52.973638 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:52.973623    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:53.056605 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.056451    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.157098 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.157069    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.257722 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.257704    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.258789 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.258775    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:28:53.258906 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.258875    2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:53.259020 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.258942    2567 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a37db1c6ee22f4efb81f40e9e9d69904-f845c31920a170ae.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-118.us-west-2.compute.internal.18e4a2f6c431686c\": read tcp 10.0.128.118:47770->3.19.45.153:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-118.us-west-2.compute.internal.18e4a2f6c431686c  default   5093 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-118.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-118.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-118.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:52 +0000 UTC,LastTimestamp:2026-11-03 20:28:52.630512667 +0000 UTC m=+0.593594653,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-118.us-west-2.compute.internal,}"
Nov 03 20:28:53.358619 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.358576    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.459088 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.459067    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.559235 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:53.559206    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.566195 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.566178    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.567143 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.567117    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.567143 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.567148    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.567290 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.567160    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.579554 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.579527    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:53 +0000 UTC" deadline="2028-08-13 22:54:32.37627246 +0000 UTC"
Nov 03 20:28:53.579554 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.579551    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15578h25m38.796723647s"
Nov 03 20:28:53.609054 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.608997    2567 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:53.619854 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.619831    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:53.688555 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.688537    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:53.690137 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.690125    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:28:53.752066 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:53.752045    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:54.062172 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.062142    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8892d1609e982d25dd2b1fad0fa8e7c5.slice/crio-70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce WatchSource:0}: Error finding container 70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce: Status 404 returned error can't find the container with id 70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce
Nov 03 20:28:54.067773 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.067757    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:28:54.107218 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.107197    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b446c071455ed0e37e47646678b6e.slice/crio-df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b WatchSource:0}: Error finding container df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b: Status 404 returned error can't find the container with id df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b
Nov 03 20:28:54.154978 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.154958    2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:54.155109 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.155094    2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:54.296523 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.296471    2567 apiserver.go:52] "Watching apiserver"
Nov 03 20:28:54.319738 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.319707    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:54.400276 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.400256    2567 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:28:54.400627 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.400607    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m7s5v","kube-system/konnectivity-agent-rqnkl","openshift-cluster-node-tuning-operator/tuned-6m9cr","openshift-multus/multus-additional-cni-plugins-l44nz","openshift-network-diagnostics/network-check-target-4gmmf","openshift-network-operator/iptables-alerter-t5zmh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz","openshift-image-registry/node-ca-njl4k","openshift-multus/multus-skrbt","kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal","openshift-dns/node-resolver-c6pqw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-8bqft","openshift-ovn-kubernetes/ovnkube-node-n42rq"]
Nov 03 20:28:54.404416 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.404400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.406682 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.406660    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.406783 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.406764    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:28:54.408935 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.408916    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.411243 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.411219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.413497 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.413477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:54.413599 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.413577    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:28:54.416507 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.416489    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.418789 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.418769    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.421597 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.421285    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.421597 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.421388    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:28:54.424922 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.424899    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.427651 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.427633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.430263 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.430473 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.430592 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.430592 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430498    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.430592 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.430704 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430595    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.430704 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldj7\" (UniqueName: \"kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.430704 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4wh\" (UniqueName: \"kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430745    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.430811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430858    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430899    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.430993    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431035 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431135    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.431438 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfp4m\" (UniqueName: \"kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431744    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.431966 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431912    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9j8j\" (UniqueName: \"kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnv7\" (UniqueName: \"kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.431961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnhr\" (UniqueName: \"kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.432001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.432055    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.432767 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.432101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.433022 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.432832    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.456967 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.456923    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:28:54.457079 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.456984    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.457079 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.456991    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.457270 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.457254    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\""
Nov 03 20:28:54.461265 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.461248    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.461380 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.461280    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\""
Nov 03 20:28:54.461380 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.461373    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.463416 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463397    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\""
Nov 03 20:28:54.463729 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463711    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.463811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463764    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:28:54.463811 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463804    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:28:54.463889 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463817    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.463932 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.463916    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:28:54.468897 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.468874    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.468971 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.468915    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.469020 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.469007    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:28:54.469064 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.469009    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\""
Nov 03 20:28:54.502460 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.502435    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:28:54.502717 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.502698    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:28:54.502940 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.502922    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:28:54.502993 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.502949    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\""
Nov 03 20:28:54.503301 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.503288    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\""
Nov 03 20:28:54.503476 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.503461    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:28:54.521610 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.521592    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:28:54.523884 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.523868    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:54.532995 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.532962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.533117 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533027    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.533117 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.533235 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnv7\" (UniqueName: \"kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.533235 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnhr\" (UniqueName: \"kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.533235 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533202    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.533376 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533255    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.533565 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533545    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.533641 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.533694 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533645    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.533805 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.533805 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.533904 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.533904 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.533904 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.534047 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4wh\" (UniqueName: \"kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.534047 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.533971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.534047 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.534208 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.534208 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.534208 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.534355 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.534355 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.534355 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.534355 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.534540 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.534540 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkfl\" (UniqueName: \"kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.534540 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.534540 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.534718 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp4m\" (UniqueName: \"kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.534718 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.534718 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.534718 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.534900 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.534900 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.534995 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.534995 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.534966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:54.535096 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.535096 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.535209 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.535287 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535265    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.535393 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.535447 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.535447 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535435    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.535549 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.535549 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535463    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.535630 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535583    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.535630 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535607    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.535715 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535654    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.535773 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535719    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.535829 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535771    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.535921 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535898    2567 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:28:54.535997 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.535907    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerStarted","Data":"70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce"}
Nov 03 20:28:54.536204 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536176    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.536548 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536504    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.536803 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536604    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.536803 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.536803 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536675    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.536803 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536584    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.536803 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.537059 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536811    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.537059 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9j8j\" (UniqueName: \"kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.537059 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536872    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.537059 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.536911    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.537445 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537307    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.537445 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.537445 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.537445 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537481    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.537523    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537562    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537584    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.537588    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.037567715 +0000 UTC m=+3.000649710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537625    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.537684 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537784    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537810    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537890    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537914    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.537923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldj7\" (UniqueName: \"kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538088    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.538186 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538170    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538303    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538360    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsfv\" (UniqueName: \"kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538391    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"1c4b446c071455ed0e37e47646678b6e","Type":"ContainerStarted","Data":"df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b"}
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538434    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538491    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.539014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538555    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538666    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538688    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538695    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrwh\" (UniqueName: \"kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538775    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538826    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538862    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538965    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.538992    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.539131    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.539149    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.539154    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.539206    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.539215    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.539648 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.539282    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.039263861 +0000 UTC m=+3.002345839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:54.540161 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.539820    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.540509 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.540493    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.561117 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.561075    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:54.561171 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.561125    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qg4tv"
Nov 03 20:28:54.566443 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566424    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:28:54.566521 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566426    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.566521 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566460    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:28:54.566635 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566542    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.566635 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566565    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.566635 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566581    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:28:54.566635 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566571    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:28:54.566833 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566653    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\""
Nov 03 20:28:54.566833 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566682    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\""
Nov 03 20:28:54.566833 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566704    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.566833 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566788    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:28:54.566950 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566839    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:28:54.566950 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.566907    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\""
Nov 03 20:28:54.594773 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.594749    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnv7\" (UniqueName: \"kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.594849 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.594759    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnhr\" (UniqueName: \"kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.595279 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.595264    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:54.595325 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.595285    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:54.595325 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.595295    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:54.595393 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:54.595361    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:55.095346075 +0000 UTC m=+3.058428073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:54.597185 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.597164    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4wh\" (UniqueName: \"kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.597457 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.597440    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp4m\" (UniqueName: \"kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.598198 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.598174    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9j8j\" (UniqueName: \"kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:54.599168 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.599150    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldj7\" (UniqueName: \"kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.639115 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639080    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639196 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.639263 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639228    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639263 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.639372 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639372 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639297    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.639372 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639368    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639410    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.639516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.639516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639538    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639606    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639539    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639647    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639696    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.639747 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639742    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639788    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639791    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsfv\" (UniqueName: \"kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639848    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639866    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639920    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrwh\" (UniqueName: \"kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639953    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.639991    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640271 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640176    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640177    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640220    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640262    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640275    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640329    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640354    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkfl\" (UniqueName: \"kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640434    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.640902 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.640470    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.642059 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.642042    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.642336 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.642318    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.701572 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.701552    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrwh\" (UniqueName: \"kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.701728 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.701711    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsfv\" (UniqueName: \"kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.701804 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.701787    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkfl\" (UniqueName: \"kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.720809 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.720790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:28:54.723024 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.723011    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:28:54.731491 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.731471    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98011d8e_1c32_46d2_ba93_b0a8ff367bc4.slice/crio-cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0 WatchSource:0}: Error finding container cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0: Status 404 returned error can't find the container with id cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0
Nov 03 20:28:54.734008 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.733989    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491877c9_e16e_405a_b84b_95db063f5b93.slice/crio-824f20fcd660644f677c2e4cfad45ff1f3858eb2161a1ea032fff5084dc62280 WatchSource:0}: Error finding container 824f20fcd660644f677c2e4cfad45ff1f3858eb2161a1ea032fff5084dc62280: Status 404 returned error can't find the container with id 824f20fcd660644f677c2e4cfad45ff1f3858eb2161a1ea032fff5084dc62280
Nov 03 20:28:54.743788 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.743774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:28:54.745272 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.745248    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-skrbt"
Nov 03 20:28:54.747366 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.747349    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:28:54.749636 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.749615    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:28:54.754285 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.754268    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f06038_b436_45b8_9502_b9e2a4ddd4bf.slice/crio-ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c WatchSource:0}: Error finding container ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c: Status 404 returned error can't find the container with id ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c
Nov 03 20:28:54.757847 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.757822    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfab6970_50da_4943_9b93_f0f4089aacb2.slice/crio-12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3 WatchSource:0}: Error finding container 12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3: Status 404 returned error can't find the container with id 12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3
Nov 03 20:28:54.762727 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.762709    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7baebc9_c7a8_4ee7_8983_4721a97edef4.slice/crio-fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213 WatchSource:0}: Error finding container fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213: Status 404 returned error can't find the container with id fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213
Nov 03 20:28:54.763648 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.763630    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc243550e_f884_41f5_986f_69456856c89f.slice/crio-3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3 WatchSource:0}: Error finding container 3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3: Status 404 returned error can't find the container with id 3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3
Nov 03 20:28:54.769183 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.769167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:28:54.771357 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.771339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:28:54.775483 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:54.775372    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:28:54.779421 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.779399    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aff89a_851d_4a9a_9b60_63c75683b36a.slice/crio-8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5 WatchSource:0}: Error finding container 8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5: Status 404 returned error can't find the container with id 8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5
Nov 03 20:28:54.785585 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.785566    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa85e6e6_4c02_4730_ac6d_ab7e8b6f82da.slice/crio-45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5 WatchSource:0}: Error finding container 45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5: Status 404 returned error can't find the container with id 45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5
Nov 03 20:28:54.786917 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:28:54.786848    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c80c5b_5844_40f9_a886_13b5c312eb44.slice/crio-0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca WatchSource:0}: Error finding container 0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca: Status 404 returned error can't find the container with id 0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca
Nov 03 20:28:55.044385 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.044307    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:55.044385 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.044354    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:55.044566 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.044475    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:55.044566 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.044532    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.044519246 +0000 UTC m=+4.007601227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:55.044566 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.044474    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:55.044670 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.044566    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.044557473 +0000 UTC m=+4.007639450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:55.144869 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.144832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:55.145032 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.144957    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:55.145032 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.144974    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:55.145032 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.144984    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:55.145185 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.145037    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.145018559 +0000 UTC m=+4.108100538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:55.529689 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.529610    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:55.530393 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:55.529755    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:28:55.542493 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.542457    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca"}
Nov 03 20:28:55.544176 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.544106    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6pqw" event={"ID":"a1aff89a-851d-4a9a-9b60-63c75683b36a","Type":"ContainerStarted","Data":"8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5"}
Nov 03 20:28:55.546869 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.546833    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqnkl" event={"ID":"c243550e-f884-41f5-986f-69456856c89f","Type":"ContainerStarted","Data":"3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3"}
Nov 03 20:28:55.548516 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.548487    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerStarted","Data":"fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213"}
Nov 03 20:28:55.550052 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.550025    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t5zmh" event={"ID":"46f06038-b436-45b8-9502-b9e2a4ddd4bf","Type":"ContainerStarted","Data":"ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c"}
Nov 03 20:28:55.552480 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.552424    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6m9cr" event={"ID":"491877c9-e16e-405a-b84b-95db063f5b93","Type":"ContainerStarted","Data":"824f20fcd660644f677c2e4cfad45ff1f3858eb2161a1ea032fff5084dc62280"}
Nov 03 20:28:55.553710 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.553666    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-njl4k" event={"ID":"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da","Type":"ContainerStarted","Data":"45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5"}
Nov 03 20:28:55.554955 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.554929    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-skrbt" event={"ID":"dfab6970-50da-4943-9b93-f0f4089aacb2","Type":"ContainerStarted","Data":"12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3"}
Nov 03 20:28:55.556176 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.556149    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz" event={"ID":"98011d8e-1c32-46d2-ba93-b0a8ff367bc4","Type":"ContainerStarted","Data":"cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0"}
Nov 03 20:28:55.566052 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.565999    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:55 +0000 UTC" deadline="2028-08-05 22:43:36.183706822 +0000 UTC"
Nov 03 20:28:55.566151 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:55.566055    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15386h14m40.617655204s"
Nov 03 20:28:56.050351 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.050317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:56.050536 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.050427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:56.050622 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.050575    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:56.050681 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.050640    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.050621092 +0000 UTC m=+6.013703083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:56.050746 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.050722    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:56.050801 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.050762    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.050748488 +0000 UTC m=+6.013830479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:56.151770 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.151730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:56.152058 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.151952    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:56.152058 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.151971    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:56.152058 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.151985    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:56.152058 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.152046    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:58.152027569 +0000 UTC m=+6.115109570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:56.532562 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.529925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:56.532562 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.530081    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:28:56.532562 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.532081    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:56.532562 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:56.532227    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:28:56.566906 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.566863    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:55 +0000 UTC" deadline="2028-08-19 23:54:11.797331404 +0000 UTC"
Nov 03 20:28:56.566906 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:56.566904    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15723h25m15.230432527s"
Nov 03 20:28:57.529384 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:57.529360    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:57.529549 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:57.529492    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.067414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.067474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.067666    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.067674    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.067731    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:29:02.067710696 +0000 UTC m=+10.030792692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:58.067787 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.067751    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:02.067740314 +0000 UTC m=+10.030822297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:58.168762 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.168386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:58.168762 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.168600    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:58.168762 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.168620    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:58.168762 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.168633    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:58.168762 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.168699    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:02.16868029 +0000 UTC m=+10.131762284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:58.530326 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.530003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:28:58.530848 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.530087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:28:58.530848 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.530644    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:28:58.530848 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:58.530509    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:28:58.602228 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.602199    2567 generic.go:356] "Generic (PLEG): container finished" podID="8892d1609e982d25dd2b1fad0fa8e7c5" containerID="861165926a981577c38b058e20b369140bb15e3704e51393e2a64236c98c26f2" exitCode=0
Nov 03 20:28:58.602327 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:58.602243    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerDied","Data":"861165926a981577c38b058e20b369140bb15e3704e51393e2a64236c98c26f2"}
Nov 03 20:28:59.529462 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.529426    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:28:59.529951 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:28:59.529586    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:28:59.609305 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.609228    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"1c4b446c071455ed0e37e47646678b6e","Type":"ContainerStarted","Data":"73c69f5caad4eaab57e4918a436b864ce87046662b277400e19bf3951a41c90f"}
Nov 03 20:28:59.614484 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.613993    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerStarted","Data":"83ebdb50ccd90dbc8e216b2be5c3596634974e2143e12f3f88bbe007325fd590"}
Nov 03 20:28:59.617939 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.617912    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="82a586279214e3909d9ed8efbcda16697009d44c6d980efbe7e84b326a7b79d5" exitCode=0
Nov 03 20:28:59.618083 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.617957    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"82a586279214e3909d9ed8efbcda16697009d44c6d980efbe7e84b326a7b79d5"}
Nov 03 20:28:59.724785 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.724724    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal" podStartSLOduration=5.724707591 podStartE2EDuration="5.724707591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:59.723929687 +0000 UTC m=+7.687011686" watchObservedRunningTime="2026-11-03 20:28:59.724707591 +0000 UTC m=+7.687789590"
Nov 03 20:28:59.840416 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:28:59.840347    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" podStartSLOduration=5.840317026 podStartE2EDuration="5.840317026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:59.839324405 +0000 UTC m=+7.802406406" watchObservedRunningTime="2026-11-03 20:28:59.840317026 +0000 UTC m=+7.803399027"
Nov 03 20:29:00.530626 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:00.530259    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:00.530626 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:00.530408    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:00.530626 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:00.530494    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:00.530626 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:00.530578    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:00.629378 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:00.628975    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqnkl" event={"ID":"c243550e-f884-41f5-986f-69456856c89f","Type":"ContainerStarted","Data":"b97db0e753297af60d1e010e59dd047f43636b14103f978747f5aa8a477a46fb"}
Nov 03 20:29:00.744824 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:00.744760    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rqnkl" podStartSLOduration=2.8498165540000002 podStartE2EDuration="7.744739068s" totalImagesPullingTime="4.894922514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:00.743685696 +0000 UTC m=+8.706767696" watchObservedRunningTime="2026-11-03 20:29:00.744739068 +0000 UTC m=+8.707821072"
Nov 03 20:29:01.529787 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:01.529749    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:01.529992 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:01.529955    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:01.633474 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:01.633381    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-njl4k" event={"ID":"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da","Type":"ContainerStarted","Data":"7708e6afb78b554a130b98d5fa3ecbe3411930d8d680de017344a47f4f903970"}
Nov 03 20:29:01.639354 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:01.639327    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz" event={"ID":"98011d8e-1c32-46d2-ba93-b0a8ff367bc4","Type":"ContainerStarted","Data":"908c485f8841b37a93195238f0884cbeeb3464dd5b6a464a3f92534f1c303395"}
Nov 03 20:29:02.101412 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:02.101312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:02.101412 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:02.101373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:02.101607 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.101518    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:02.101607 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.101576    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:29:10.101558803 +0000 UTC m=+18.064640795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:02.101719 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.101639    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:02.101719 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.101673    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:10.101663586 +0000 UTC m=+18.064745577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:02.202109 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:02.202074    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:02.202291 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.202218    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:29:02.202291 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.202246    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:29:02.202291 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.202261    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:02.202452 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.202319    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:10.202301095 +0000 UTC m=+18.165383093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:02.530934 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:02.530859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:02.531119 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.530980    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:02.531119 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:02.531094    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:02.531323 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:02.531216    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:03.438369 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.438346    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:03.439232 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.439211    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:03.529841 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.529814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:03.529962 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:03.529920    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:03.556091 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.555781    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-njl4k" podStartSLOduration=4.473757718 podStartE2EDuration="10.555760716s" totalImagesPullingTime="6.082002998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.753050576 +0000 UTC m=+9.716132577" watchObservedRunningTime="2026-11-03 20:29:03.555760716 +0000 UTC m=+11.518842717"
Nov 03 20:29:03.633840 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.633817    2567 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:29:03.650127 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.650073    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-skrbt" event={"ID":"dfab6970-50da-4943-9b93-f0f4089aacb2","Type":"ContainerStarted","Data":"ef9e07e7d98e2aa959af08a6733ad7b2c300fbb5f5c8ba933da634e2d91a1958"}
Nov 03 20:29:03.664827 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.664796    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz" event={"ID":"98011d8e-1c32-46d2-ba93-b0a8ff367bc4","Type":"ContainerStarted","Data":"58e0390ba82b7a3fead3d76f88e3acdb52a9eab72542aad59123e25badb97a81"}
Nov 03 20:29:03.769333 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:03.769286    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-skrbt" podStartSLOduration=2.08814975 podStartE2EDuration="10.769270334s" totalImagesPullingTime="8.681120584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:03.768478335 +0000 UTC m=+11.731560336" watchObservedRunningTime="2026-11-03 20:29:03.769270334 +0000 UTC m=+11.732352333"
Nov 03 20:29:04.403435 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.403353    2567 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:29:03.633834866Z","UUID":"5fe8418e-196b-41ad-99ee-e0480188c04c","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:29:04.405016 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.404982    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:29:04.405016 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.405014    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:29:04.529605 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.529351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:04.529989 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:04.529706    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:04.529989 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.529867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:04.529989 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:04.529958    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:04.668861 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.668782    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6pqw" event={"ID":"a1aff89a-851d-4a9a-9b60-63c75683b36a","Type":"ContainerStarted","Data":"048b0f70a73b97c5c875b12d074b3d09c6473551d30f5678942f409164845ac6"}
Nov 03 20:29:04.672487 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.671986    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="b03e30505cb8cf86a2ec415eb1d5f84ee7de674940dcda1e48b6f388b83a5c3c" exitCode=0
Nov 03 20:29:04.672487 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.672086    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"b03e30505cb8cf86a2ec415eb1d5f84ee7de674940dcda1e48b6f388b83a5c3c"}
Nov 03 20:29:04.785798 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:04.785358    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c6pqw" podStartSLOduration=2.809243459 podStartE2EDuration="11.785340677s" totalImagesPullingTime="8.976097218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:04.785017566 +0000 UTC m=+12.748099566" watchObservedRunningTime="2026-11-03 20:29:04.785340677 +0000 UTC m=+12.748422678"
Nov 03 20:29:05.529139 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:05.529113    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:05.529314 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:05.529243    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:06.529252 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.529219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:06.529708 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.529268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:06.529708 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:06.529357    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:06.529708 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:06.529447    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:06.678529 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.678454    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="b74ee7447d48329b3d29755f3c4085f1961a17bf490df92de0bc9325b1dae511" exitCode=0
Nov 03 20:29:06.678683 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.678550    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"b74ee7447d48329b3d29755f3c4085f1961a17bf490df92de0bc9325b1dae511"}
Nov 03 20:29:06.680611 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.680576    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t5zmh" event={"ID":"46f06038-b436-45b8-9502-b9e2a4ddd4bf","Type":"ContainerStarted","Data":"37215a7719afe1069c3389f3f97c7aaf016666fefac5cdd5b07f45ec48f93f8e"}
Nov 03 20:29:06.682314 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.682284    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6m9cr" event={"ID":"491877c9-e16e-405a-b84b-95db063f5b93","Type":"ContainerStarted","Data":"9e6c414c18d1c9b34433a6228e28cf3be6e620c7ee753e8ea88b52d0b43302a3"}
Nov 03 20:29:06.898930 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:06.898879    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t5zmh" podStartSLOduration=4.598893406 podStartE2EDuration="13.898865304s" totalImagesPullingTime="9.299971898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:06.898503155 +0000 UTC m=+14.861585158" watchObservedRunningTime="2026-11-03 20:29:06.898865304 +0000 UTC m=+14.861947300"
Nov 03 20:29:07.008430 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:07.008334    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6m9cr" podStartSLOduration=2.452479566 podStartE2EDuration="14.008320224s" totalImagesPullingTime="11.555840658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:07.007681041 +0000 UTC m=+14.970763041" watchObservedRunningTime="2026-11-03 20:29:07.008320224 +0000 UTC m=+14.971402220"
Nov 03 20:29:07.530009 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:07.529982    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:07.530460 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:07.530115    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:08.530079 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.529991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:08.530079 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.530030    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:08.530817 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:08.530147    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:08.530817 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:08.530277    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:08.688946 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.688911    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz" event={"ID":"98011d8e-1c32-46d2-ba93-b0a8ff367bc4","Type":"ContainerStarted","Data":"a556493207ceb45257e29d39deddc5877a4914a99c9c7500d33171c6303921f1"}
Nov 03 20:29:08.691530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.691505    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="83116649adc7234057a2d2d7d35ebae7009b3b0b7fa42fcc12dc0b9dc213be85" exitCode=0
Nov 03 20:29:08.691648 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.691551    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"83116649adc7234057a2d2d7d35ebae7009b3b0b7fa42fcc12dc0b9dc213be85"}
Nov 03 20:29:08.799340 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:08.799243    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz" podStartSLOduration=3.183898261 podStartE2EDuration="15.799228276s" totalImagesPullingTime="12.615330015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:08.798706297 +0000 UTC m=+16.761788308" watchObservedRunningTime="2026-11-03 20:29:08.799228276 +0000 UTC m=+16.762310271"
Nov 03 20:29:09.529968 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:09.529739    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:09.530145 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:09.530085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:10.163268 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.163237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:10.163268 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.163273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:10.163507 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.163361    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:10.163507 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.163365    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:10.163507 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.163409    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:29:26.163398101 +0000 UTC m=+34.126480076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:10.163507 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.163421    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret podName:038206a5-8ba4-456e-8e71-db0ec9cdd46c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:26.163415949 +0000 UTC m=+34.126497923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret") pod "global-pull-secret-syncer-m7s5v" (UID: "038206a5-8ba4-456e-8e71-db0ec9cdd46c") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:10.264251 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.264219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:10.264389 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.264353    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:29:10.264389 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.264369    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:29:10.264389 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.264378    2567 projected.go:196] Error preparing data for projected volume kube-api-access-kgbgj for pod openshift-network-diagnostics/network-check-target-4gmmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:10.264490 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.264425    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj podName:db4e1dcf-0e87-4463-a338-10fb9cf47d1a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:26.264409558 +0000 UTC m=+34.227491550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kgbgj" (UniqueName: "kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj") pod "network-check-target-4gmmf" (UID: "db4e1dcf-0e87-4463-a338-10fb9cf47d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:10.529452 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.529383    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:10.529588 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.529384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:10.529588 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.529492    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:10.529669 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:10.529582    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:10.697452 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.697430    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"7b570881585a6bc2cba57a900dcb7af4273cae876de1f8e64420759696a71f3c"}
Nov 03 20:29:10.697733 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.697512    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"4c8945dd8bd6321ab2e498b33de77767b4903e287e6d8ea01b3949d799626d42"}
Nov 03 20:29:10.697733 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:10.697541    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"0f9ae456f9ef9c4513ca5e65418e769c9e8637ff257ef457de0db0690097c687"}
Nov 03 20:29:11.529380 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:11.529350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:11.529533 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:11.529469    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:11.702530 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:11.702502    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"3232054ffe1c645a2c7c3d408f024447ee5deeda7c9a3806a060a5b0317a0f25"}
Nov 03 20:29:11.702893 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:11.702582    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"8a21eac0053c244c7c57255c5a899ecc0706c461497d24949a207cd17fb7ccb1"}
Nov 03 20:29:11.702893 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:11.702633    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"8831b4856e29cd41351036d43e18c78f3e5940c3d0c67d06a68f3da679cd9297"}
Nov 03 20:29:12.530414 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:12.530389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:12.530555 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:12.530472    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:12.530608 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:12.530568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:12.530674 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:12.530656    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:13.529539 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:13.529507    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:13.530013 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:13.529627    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:13.709132 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:13.709104    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"178a2d88c035e02024d4aea2f5c6eff221153b388662c9ccb744026cacb46cc8"}
Nov 03 20:29:14.529465 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:14.529433    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:14.529465 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:14.529449    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:14.529899 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:14.529537    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:14.529899 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:14.529657    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:14.714672 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:14.714644    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerStarted","Data":"907640b104195607736ad7a683e9187666d5735af25892a9b7987c2ebf262602"}
Nov 03 20:29:15.529315 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.529290    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:15.529446 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:15.529407    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:15.720745 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.720710    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" event={"ID":"f7c80c5b-5844-40f9-a886-13b5c312eb44","Type":"ContainerStarted","Data":"4594ab447df39eedfdb81df49b307dc1637a4cfd3d7dec41b58327864a195864"}
Nov 03 20:29:15.721570 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.720932    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:15.723538 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.723515    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="907640b104195607736ad7a683e9187666d5735af25892a9b7987c2ebf262602" exitCode=0
Nov 03 20:29:15.723659 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.723557    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"907640b104195607736ad7a683e9187666d5735af25892a9b7987c2ebf262602"}
Nov 03 20:29:15.739152 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.739133    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:15.887211 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:15.887151    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq" podStartSLOduration=7.143418092 podStartE2EDuration="22.88713371s" totalImagesPullingTime="15.743715618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:15.885724576 +0000 UTC m=+23.848806571" watchObservedRunningTime="2026-11-03 20:29:15.88713371 +0000 UTC m=+23.850215709"
Nov 03 20:29:16.529366 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.529336    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:16.529366 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.529365    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:16.529672 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:16.529439    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:16.529672 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:16.529550    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:16.728318 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.728286    2567 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="8cb034d263b4210372f538677dfaa6d2f17d801eb38045ed8351bf8e2f1744d9" exitCode=0
Nov 03 20:29:16.728698 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.728367    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerDied","Data":"8cb034d263b4210372f538677dfaa6d2f17d801eb38045ed8351bf8e2f1744d9"}
Nov 03 20:29:16.729014 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.728987    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:16.729399 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.729086    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:16.744657 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:16.744505    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:17.529396 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.529367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:17.529562 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:17.529497    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:17.725571 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.725510    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7s5v"]
Nov 03 20:29:17.725706 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.725596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:17.725706 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:17.725685    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:17.728482 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.728455    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gmmf"]
Nov 03 20:29:17.728830 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.728547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:17.728830 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:17.728627    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:17.728981 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.728951    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bqft"]
Nov 03 20:29:17.737245 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.737227    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:17.737361 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.737268    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l44nz" event={"ID":"f7baebc9-c7a8-4ee7-8983-4721a97edef4","Type":"ContainerStarted","Data":"3d05882a7d378d5be05beaebf0ac72575c90cd1c6f0b2af081e7d7cb413614b5"}
Nov 03 20:29:17.737361 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:17.737319    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:17.850997 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:17.850953    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l44nz" podStartSLOduration=7.473494742 podStartE2EDuration="24.850941852s" totalImagesPullingTime="17.37744711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:17.850379366 +0000 UTC m=+25.813461362" watchObservedRunningTime="2026-11-03 20:29:17.850941852 +0000 UTC m=+25.814023847"
Nov 03 20:29:19.529210 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:19.529181    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:19.529613 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:19.529222    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:19.529613 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:19.529291    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:19.529613 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:19.529294    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:19.529613 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:19.529392    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:19.529613 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:19.529466    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:21.529820 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.529797    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:21.530209 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.529818    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:21.530209 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:21.529916    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8bqft" podUID="7858d6a6-5909-4b5a-a5e1-fe41729234ec"
Nov 03 20:29:21.530209 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:21.529947    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7s5v" podUID="038206a5-8ba4-456e-8e71-db0ec9cdd46c"
Nov 03 20:29:21.530209 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.529986    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:21.530209 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:21.530075    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gmmf" podUID="db4e1dcf-0e87-4463-a338-10fb9cf47d1a"
Nov 03 20:29:21.865174 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.865108    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:21.865301 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.865238    2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:29:21.865636 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:21.865624    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:23.173879 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.173726    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:29:23.174215 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.173962    2567 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:29:23.529414 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.529387    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:23.529564 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.529423    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:23.529564 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.529556    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:23.584141 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.584111    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\""
Nov 03 20:29:23.584141 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.584124    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\""
Nov 03 20:29:23.630193 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.630164    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\""
Nov 03 20:29:23.630193 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.630197    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:23.630396 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.630217    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:23.630396 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.630265    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:29:23.630396 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:23.630295    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:29:26.013977 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.013953    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c6pqw_a1aff89a-851d-4a9a-9b60-63c75683b36a/dns-node-resolver/0.log"
Nov 03 20:29:26.176216 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.176187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:26.176353 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.176225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:26.176353 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:26.176320    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 20:29:26.176417 ip-10-0-128-118 kubenswrapper[2567]: E1103 20:29:26.176377    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs podName:7858d6a6-5909-4b5a-a5e1-fe41729234ec nodeName:}" failed. No retries permitted until 2026-11-03 20:29:58.176358944 +0000 UTC m=+66.139440931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs") pod "network-metrics-daemon-8bqft" (UID: "7858d6a6-5909-4b5a-a5e1-fe41729234ec") : secret "metrics-daemon-secret" not found
Nov 03 20:29:26.179842 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.179813    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:26.241608 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.241581    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:26.277278 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.277224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:26.279549 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.279529    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:26.548009 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.547931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:26.715906 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.715876    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7s5v"]
Nov 03 20:29:26.719017 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:29:26.718991    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038206a5_8ba4_456e_8e71_db0ec9cdd46c.slice/crio-65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4 WatchSource:0}: Error finding container 65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4: Status 404 returned error can't find the container with id 65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4
Nov 03 20:29:26.756446 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.756418    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m7s5v" event={"ID":"038206a5-8ba4-456e-8e71-db0ec9cdd46c","Type":"ContainerStarted","Data":"65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4"}
Nov 03 20:29:26.925520 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:26.925443    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gmmf"]
Nov 03 20:29:26.928730 ip-10-0-128-118 kubenswrapper[2567]: W1103 20:29:26.928704    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4e1dcf_0e87_4463_a338_10fb9cf47d1a.slice/crio-10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3 WatchSource:0}: Error finding container 10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3: Status 404 returned error can't find the container with id 10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3
Nov 03 20:29:27.436770 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:27.436746    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-njl4k_fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da/node-ca/0.log"
Nov 03 20:29:27.760168 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:27.760136    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gmmf" event={"ID":"db4e1dcf-0e87-4463-a338-10fb9cf47d1a","Type":"ContainerStarted","Data":"10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3"}
Nov 03 20:29:33.776339 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:33.776299    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gmmf" event={"ID":"db4e1dcf-0e87-4463-a338-10fb9cf47d1a","Type":"ContainerStarted","Data":"f696e33fd9dde95ffd070299feaee6b44f84d250758aed85530ae2858cb1ac3c"}
Nov 03 20:29:33.776809 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:33.776450    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:33.884258 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:33.884198    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gmmf" podStartSLOduration=34.992424326 podStartE2EDuration="40.884178014s" totalImagesPullingTime="5.891753688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:33.883564108 +0000 UTC m=+41.846646104" watchObservedRunningTime="2026-11-03 20:29:33.884178014 +0000 UTC m=+41.847260012"
Nov 03 20:29:35.021790 ip-10-0-128-118 kubenswrapper[2567]: I1103 20:29:35.021704    2567 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:29:35.021868 ip-10-0-128-118 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:29:35.029603 ip-10-0-128-118 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:29:35.029769 ip-10-0-128-118 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:29:35.029813 ip-10-0-128-118 systemd[1]: kubelet.service: Consumed 2.102s CPU time.
Nov 03 20:29:35.044569 ip-10-0-128-118 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:29:35.129908 ip-10-0-128-118 kubenswrapper[4274]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:35.129908 ip-10-0-128-118 kubenswrapper[4274]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:29:35.129908 ip-10-0-128-118 kubenswrapper[4274]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:35.129908 ip-10-0-128-118 kubenswrapper[4274]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:29:35.132011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.131930    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.132011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132011    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132015    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132018    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132022    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132025    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132027    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132030    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132046    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132049    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132052    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132054    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132058    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132061    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132063    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132066    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132070    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132072    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132075    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132077    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.132098 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132080    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132082    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132085    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132087    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132090    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132093    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132095    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132098    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132100    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132103    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132106    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132109    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132111    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132114    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132117    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132119    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132122    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132125    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132127    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.132565 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132130    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132132    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132135    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132138    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132157    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132161    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132164    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132166    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132169    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132171    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132174    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132178    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132180    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132184    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132186    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132189    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132192    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132195    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132197    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132200    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.133061 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132202    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132205    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132207    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132210    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132213    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132215    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132218    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132220    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132223    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132225    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132229    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132231    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132236    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132240    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132243    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132246    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132249    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132252    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132254    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.133567 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132257    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132260    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132263    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132265    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132268    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132271    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132274    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132277    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132280    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132282    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132285    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132287    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132290    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132292    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132297    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132301    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132304    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132307    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132310    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.134018 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132313    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132315    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132318    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132320    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132323    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132326    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132329    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132332    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.132335    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133107    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133116    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133119    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133122    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133124    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133127    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133132    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133137    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133141    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133144    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133147    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.134529 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133150    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133152    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133156    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133160    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133163    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133166    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133168    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133171    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133173    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133176    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133178    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133181    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133183    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133186    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133189    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133191    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133194    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133197    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133199    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.135013 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133202    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133204    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133213    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133216    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133219    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133221    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133224    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133227    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133229    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133232    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133234    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133237    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133240    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133242    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133245    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133247    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133250    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133252    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133255    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133257    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.135489 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133260    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133262    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133266    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133269    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133272    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133274    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133277    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133279    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133282    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133284    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133287    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133289    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133292    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133294    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133297    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133305    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133307    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133310    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133313    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133315    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.136011 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133318    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133321    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133323    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133326    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133328    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133331    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133333    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133336    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133339    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133341    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133344    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133346    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133349    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133351    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133354    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133356    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133359    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133361    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133364    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133366    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.136525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133369    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133371    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133374    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133377    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133380    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133382    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133385    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133387    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133395    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133398    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133404    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133406    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133409    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133411    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133414    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133417    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133919    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133925    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133928    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.137005 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133931    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133934    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133936    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133939    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133942    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133944    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133947    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133949    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133952    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133954    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133957    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133960    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133963    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133966    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133969    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133971    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133974    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133977    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133979    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.137506 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133982    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133985    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133988    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133991    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133994    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.133997    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134000    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134004    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134007    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134010    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134013    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134016    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134019    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134022    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134024    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134027    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134030    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134051    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134055    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.137991 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134058    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134061    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134065    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134067    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134071    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134074    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134077    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134079    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134082    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134084    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134087    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134090    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134093    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134095    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134098    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134101    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134105    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134107    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134110    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134112    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.138471 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134115    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134118    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134120    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134123    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134125    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134128    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134130    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134133    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134135    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134137    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134140    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134143    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134145    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134148    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134151    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134153    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134156    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134159    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134162    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.138975 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134164    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134167    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134169    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134172    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134175    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134178    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134180    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134183    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134185    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134188    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134190    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134193    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134196    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134198    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134201    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134203    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134206    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134208    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134211    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134213    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.139447 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134216    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134219    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134222    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134225    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134227    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134230    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134327    4274 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134338    4274 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134345    4274 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134350    4274 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134354    4274 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134357    4274 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134361    4274 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134366    4274 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134369    4274 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134372    4274 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134376    4274 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134379    4274 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134382    4274 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134385    4274 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134388    4274 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134391    4274 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134394    4274 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:29:35.139936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134397    4274 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134401    4274 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134405    4274 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134408    4274 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134411    4274 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134417    4274 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134421    4274 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134425    4274 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134428    4274 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134432    4274 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134435    4274 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134439    4274 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134442    4274 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134446    4274 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134449    4274 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134453    4274 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134456    4274 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134459    4274 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134462    4274 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134465    4274 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134468    4274 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134472    4274 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134475    4274 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134478    4274 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:29:35.140613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134482    4274 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134485    4274 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134489    4274 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134492    4274 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134495    4274 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134498    4274 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134501    4274 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134505    4274 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134508    4274 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134511    4274 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134522    4274 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134525    4274 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134528    4274 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134534    4274 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134537    4274 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134541    4274 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134545    4274 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134548    4274 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134551    4274 flags.go:64] FLAG: --help="false"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134555    4274 flags.go:64] FLAG: --hostname-override="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134559    4274 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134562    4274 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134565    4274 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:29:35.141598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134568    4274 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134572    4274 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134575    4274 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134577    4274 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134580    4274 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134583    4274 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134586    4274 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134589    4274 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134592    4274 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134595    4274 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134598    4274 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134601    4274 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134603    4274 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134606    4274 flags.go:64] FLAG: --lock-file=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134609    4274 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134612    4274 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134615    4274 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134625    4274 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134628    4274 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134631    4274 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134634    4274 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134637    4274 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134642    4274 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134645    4274 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134648    4274 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:29:35.142603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134652    4274 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134655    4274 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134659    4274 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134664    4274 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134667    4274 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134670    4274 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134672    4274 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134675    4274 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134679    4274 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134681    4274 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134688    4274 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134691    4274 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134694    4274 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134697    4274 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134700    4274 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134703    4274 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134706    4274 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134709    4274 flags.go:64] FLAG: --port="10250"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134712    4274 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134715    4274 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-05edf57cb66b72027"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134718    4274 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134721    4274 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134724    4274 flags.go:64] FLAG: --register-node="true"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134727    4274 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134731    4274 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134734    4274 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:29:35.143671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134737    4274 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134740    4274 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134744    4274 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134748    4274 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134751    4274 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134754    4274 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134757    4274 flags.go:64] FLAG: --runonce="false"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134760    4274 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134763    4274 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134768    4274 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134771    4274 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134773    4274 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134776    4274 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134779    4274 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134783    4274 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134786    4274 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134789    4274 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134792    4274 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134794    4274 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134798    4274 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134801    4274 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134803    4274 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134809    4274 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134811    4274 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134814    4274 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:29:35.144856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134818    4274 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134821    4274 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134824    4274 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134827    4274 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134829    4274 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134832    4274 flags.go:64] FLAG: --v="2"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134836    4274 flags.go:64] FLAG: --version="false"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134841    4274 flags.go:64] FLAG: --vmodule=""
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134844    4274 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.134848    4274 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134976    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134979    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134983    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134986    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134989    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134992    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134996    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.134999    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135001    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135004    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135007    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.145905 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135009    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135012    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135015    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135017    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135020    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135023    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135026    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135029    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135044    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135047    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135050    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135053    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135056    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135058    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135061    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135063    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135066    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135068    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135071    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.146823 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135073    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135076    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135081    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135084    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135087    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135089    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135092    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135094    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135099    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135101    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135104    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135107    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135109    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135112    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135115    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135117    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135120    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135122    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135125    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135127    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.147536 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135130    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135132    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135135    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135139    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135142    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135145    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135148    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135150    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135153    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135158    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135162    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135167    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135171    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135175    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135180    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135185    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135189    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135192    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135195    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.148106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135198    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135202    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135205    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135208    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135212    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135216    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135218    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135221    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135224    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135227    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135230    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135233    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135235    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135238    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135240    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135243    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135246    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135248    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135251    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.148754 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135253    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135256    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135259    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135261    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135264    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135266    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135269    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135272    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135277    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135280    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135283    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135286    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135289    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135291    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135295    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135297    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135300    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.149319 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135303    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.135330    4274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135365    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135369    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135373    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135376    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135378    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135381    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135384    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135386    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135389    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135391    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135394    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135396    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135399    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135402    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.149765 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135405    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135408    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135413    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135416    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135419    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135422    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135425    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135428    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135432    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135435    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135438    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135441    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135444    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135448    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135450    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135453    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135455    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135458    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135461    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135463    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.150365 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135466    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135468    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135471    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135473    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135476    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135478    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135481    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135484    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135487    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135489    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135492    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135495    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135497    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135500    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135503    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135505    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135508    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135510    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135512    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.151072 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135515    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135518    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135521    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135523    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135526    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135529    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135533    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135536    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135538    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135541    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135543    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135547    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135549    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135552    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135554    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135557    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135559    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135562    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135564    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135567    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.151812 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135570    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135572    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135575    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135577    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135580    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135582    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135584    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135587    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135590    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135592    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135595    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135598    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135600    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135603    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135605    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135608    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135610    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135613    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135617    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135620    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.152671 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135623    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135625    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135627    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135630    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135632    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135635    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135637    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135640    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135643    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135646    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135649    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135652    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.135654    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.139742    4274 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.139759    4274 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.139778    4274 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.139782    4274 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139844    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139848    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.153568 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139852    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139855    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139858    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139861    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139863    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139866    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139869    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139871    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139874    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139876    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139879    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139882    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139884    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139887    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139891    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139893    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139896    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139899    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139901    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.154493 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139905    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139907    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139910    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139913    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139915    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139918    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139921    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139923    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139933    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139937    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139940    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139942    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139945    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139948    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139951    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139953    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139956    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139958    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139961    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139964    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.155338 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139966    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139968    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139971    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139973    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139976    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139978    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139981    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139984    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139987    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139989    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139992    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139995    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.139997    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140000    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140003    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140005    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140010    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140014    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140017    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.156266 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140020    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140023    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140044    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140050    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140053    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140057    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140060    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140062    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140065    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140068    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140071    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140074    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140076    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140079    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140081    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140084    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140086    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140089    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140091    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.157136 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140094    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140097    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140099    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140102    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140105    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140107    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140110    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140113    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140116    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140118    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140121    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140124    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140126    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140129    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140131    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140134    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140143    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140146    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140148    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140151    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.157996 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140153    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140155    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140158    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140160    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140163    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140165    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.140168    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.140196    4274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141089    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141107    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141112    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141117    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141122    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141126    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141133    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141141    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.158932 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141146    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141150    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141161    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141165    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141169    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141174    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141178    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141182    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141186    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141191    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141195    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141199    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141203    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141236    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141246    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141251    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141255    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141259    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141263    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.159681 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141267    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141272    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141276    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141280    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141285    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141289    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141293    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141297    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141306    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141311    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141315    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141319    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141323    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141328    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141332    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141336    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141340    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141344    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141348    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141352    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.160646 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141356    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141366    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141370    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141374    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141378    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141381    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141386    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141419    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141423    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141428    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141431    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141435    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141439    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141446    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141455    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141509    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141516    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141546    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141552    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141558    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.161525 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141563    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141567    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141571    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141575    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141579    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141584    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141588    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141597    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141602    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141606    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141611    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141615    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141620    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141625    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141629    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141631    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141727    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141858    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141867    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.162348 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141872    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141876    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141881    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141885    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141890    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141896    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141901    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141905    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141910    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141914    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141920    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141925    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141929    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141933    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141940    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141944    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141950    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141954    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141958    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.162899 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.141963    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.142014    4274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142105    4274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142116    4274 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142121    4274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142126    4274 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142131    4274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142135    4274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142139    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142145    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142149    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142154    4274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142158    4274 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142163    4274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142168    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142172    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:29:35.163629 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142177    4274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142182    4274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142186    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142193    4274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142200    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142206    4274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142211    4274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142216    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142220    4274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142225    4274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142230    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142234    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142238    4274 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142242    4274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142246    4274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142251    4274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142255    4274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142260    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142265    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:29:35.164305 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142269    4274 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142274    4274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142278    4274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142282    4274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142286    4274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142291    4274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142295    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142300    4274 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142304    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142309    4274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142313    4274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142317    4274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142322    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142326    4274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142331    4274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142335    4274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142340    4274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142344    4274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142349    4274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:29:35.165095 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142353    4274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142357    4274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142362    4274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142366    4274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142370    4274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142375    4274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142380    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142384    4274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142388    4274 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142393    4274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142397    4274 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142402    4274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142406    4274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142411    4274 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142416    4274 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142420    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142425    4274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142429    4274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142434    4274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142438    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:29:35.165670 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142442    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142447    4274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142452    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142460    4274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142466    4274 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142471    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142477    4274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142482    4274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142487    4274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142491    4274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142496    4274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142500    4274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142504    4274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142509    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142520    4274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142525    4274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142529    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142533    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142537    4274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:29:35.166246 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142542    4274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142547    4274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142552    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142556    4274 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142561    4274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142565    4274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142570    4274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142574    4274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142579    4274 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142583    4274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142587    4274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142594    4274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142598    4274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142603    4274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:35.142606    4274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.142693    4274 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144258    4274 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144320    4274 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144579    4274 server.go:1014] "Starting client certificate rotation"
Nov 03 20:29:35.166698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144711    4274 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144766    4274 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:53 +0000 UTC" deadline="2028-07-11 11:37:41.061577288 +0000 UTC"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.144793    4274 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14775h8m5.916786105s"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.145299    4274 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.146983    4274 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.148500    4274 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.149974    4274 log.go:25] "Validated CRI v1 image API"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.150780    4274 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:29:35.167179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.152647    4274 fs.go:135] Filesystem UUIDs: map[3751f397-fdf1-476d-aaaa-0bf8d0afb561:/dev/nvme0n1p4 5585f9de-dc3d-41c8-b742-97bb23eb3e98:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.152673    4274 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/038206a5-8ba4-456e-8e71-db0ec9cdd46c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/038206a5-8ba4-456e-8e71-db0ec9cdd46c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46f06038-b436-45b8-9502-b9e2a4ddd4bf/volumes/kubernetes.io~projected/kube-api-access-fc4wh:{mountpoint:/var/lib/kubelet/pods/46f06038-b436-45b8-9502-b9e2a4ddd4bf/volumes/kubernetes.io~projected/kube-api-access-fc4wh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~projected/kube-api-access-zdnv7:{mountpoint:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~projected/kube-api-access-zdnv7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7858d6a6-5909-4b5a-a5e1-fe41729234ec/volumes/kubernetes.io~projected/kube-api-access-x9j8j:{mountpoint:/var/lib/kubelet/pods/7858d6a6-5909-4b5a-a5e1-fe41729234ec/volumes/kubernetes.io~projected/kube-api-access-x9j8j major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98011d8e-1c32-46d2-ba93-b0a8ff367bc4/volumes/kubernetes.io~projected/kube-api-access-rfp4m:{mountpoint:/var/lib/kubelet/pods/98011d8e-1c32-46d2-ba93-b0a8ff367bc4/volumes/kubernetes.io~projected/kube-api-access-rfp4m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1aff89a-851d-4a9a-9b60-63c75683b36a/volumes/kubernetes.io~projected/kube-api-access-crsfv:{mountpoint:/var/lib/kubelet/pods/a1aff89a-851d-4a9a-9b60-63c75683b36a/volumes/kubernetes.io~projected/kube-api-access-crsfv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c243550e-f884-41f5-986f-69456856c89f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c243550e-f884-41f5-986f-69456856c89f/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db4e1dcf-0e87-4463-a338-10fb9cf47d1a/volumes/kubernetes.io~projected/kube-api-access-kgbgj:{mountpoint:/var/lib/kubelet/pods/db4e1dcf-0e87-4463-a338-10fb9cf47d1a/volumes/kubernetes.io~projected/kube-api-access-kgbgj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfab6970-50da-4943-9b93-f0f4089aacb2/volumes/kubernetes.io~projected/kube-api-access-9ldj7:{mountpoint:/var/lib/kubelet/pods/dfab6970-50da-4943-9b93-f0f4089aacb2/volumes/kubernetes.io~projected/kube-api-access-9ldj7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7baebc9-c7a8-4ee7-8983-4721a97edef4/volumes/kubernetes.io~projected/kube-api-access-6xnhr:{mountpoint:/var/lib/kubelet/pods/f7baebc9-c7a8-4ee7-8983-4721a97edef4/volumes/kubernetes.io~projected/kube-api-access-6xnhr major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~projected/kube-api-access-4mkfl:{mountpoint:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~projected/kube-api-access-4mkfl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da/volumes/kubernetes.io~projected/kube-api-access-shrwh:{mountpoint:/var/lib/kubelet/pods/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da/volumes/kubernetes.io~projected/kube-api-access-shrwh major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ecd1d7a8900f020b72f931c34037087ddba3feb1e822d9ee00814c219fe2f430/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/dcda396eab9d5f60d02cd6c5450369f42419a0222672a88437f73f1bf99774b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0dfdf5c5278335590c476b0d0301f714c5d3bb1573d7b890fee8cc69f2fdbca3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1b1969ff9e18ba269b8f3a76a18a772a9659c4c38292aeee150c5fb2f52398d3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/f429809c8ae65af478762067160982061ac70b7f86f5aa759cfaf03b1d059163/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8cab24675bcc058965c9e47d678356c847f4d836ae659cf3b11c8b04a80e14e8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a7f15d0e1eb696eb7c47539a888fc8c28a9ae8f2435105b62d43fc2faa149421/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8060f4097ded5e7aae6519114141b316896b3a8e50c830b720e7b5ba9c3fcc20/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/56ca586b719af134ca5034f379b1637cf9aecf58e3cd2a33a93c1e68bd66b603/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2d8f09afe710a2a131e123db369df8099be3b832b96891d2e22e36f5119d40bd/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a5d716779187a295de7239d63e6b0f901c408d52b3030460e8ba12ca39b98ca6/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f1e361cbe97ae849aeaa10f1474ff66f3107cec525ec560e5b93455ec8c4ff57/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/bf147703c5757637db9d16fb570f6e214fbe849da649d2f905b0708035565de9/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c8f6b9d40a153ff48331005ddf567a87d0bc7cde35ec8811acfbac41e5755f90/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bee92d1f16237210a03c04f04061eb9da06ac58daeb78d477f2aeae0b89e28dc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3784353abe7d7a300279c00ef092b03feefff92988b42bda035338cb54f9b930/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a66a9d7d72509d836c003564ea2ad18579b90ac38ccf817eca78951b93a7fc9e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/48714d63e3f250699684c5eec6c00c645947e88840e6999c00d922c2c6461acb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3339bfe44a74d9923dde425a5c4e0eb2462b5bf5ff2f7d79c549a490e8d3d339/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d61045e40009255efdac3fcea07e07830cd05540eb7a68872610aa084d58627d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/42356fc56c49b9b02915f5bc39d3c70d918bbbbd19c6eefdc028d445404e424f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/05477a98600075ec98461d5ff10366de898ad9312ac983cb15f4cb64b15cb5f3/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/cf3166549a7582a804b254ea2397cf9f0f1950aa87276213e96c329633be0dc8/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/bc1b1ea908f0aca6e9c98eb3efecb2bc863300d5d082c632afa010c645c91eef/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a7925cc9a4aeacbc1ad3cdd55c7462b0a7a0174c9db16f60944c322eee7c7119/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/001121c5a68df64c1aaa0fd365940733e00c0788166386d761267e93aff24302/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a85211e29dc75e22768663dcba11f6815fd5f410cba53eff6ca1dcf5fae5ede6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0c27e9d65b402d9692dc3f0ea1e717015bd199ca9725c35d2cc0552f518d629b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d291178cef050fdb12798e75e064d94db2705793dfb5052b65e3128e38200127/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a9e7f6e6cce984cb017ad70345955c3a4b275e2461b1e1fac61a063767d37ade/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a81632f655f0a428c563548e7d5e58dd9db767df82661c3863c79095fcf3f62d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/b2e87eda9fd781e4187b1fb14f4a47432547260b3d1a9b0614743535c7986258/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a8b02b2cb9f1179353a3771b265a4aca82998508556ee717063d94959bd8fc7d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/984c7c3c6d4fa779ef297d5766ce3588cbd092c3700fb490057b52d0d19586d1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/122b30a21f59b2e2d12b1087ceeea4f794fb937c97df03f3569ffd5b177ceaa6/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.161770    4274 manager.go:217] Machine: {Timestamp:2026-11-03 20:29:35.160947749 +0000 UTC m=+0.087517998 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099491 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d6b64f3c194930a49e6efe8f8165 SystemUUID:ec27d6b6-4f3c-1949-30a4-9e6efe8f8165 BootID:8150a4b3-d76a-4ec7-9356-1c3a9092f457 Filesystems:[{Device:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~projected/kube-api-access-4mkfl DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfc63aecb4042170834e4d65b3168be716100d9a6b75b94bf993603a9fae30f0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0074c95c44a1cfdf6ada6aed732294da7bd98671f512ca50b180846f7d69a3ca/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12993ee6f22635dfce6014c4b9efca763e780f725b43cc38fb3b7f76556c8ce3/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7baebc9-c7a8-4ee7-8983-4721a97edef4/volumes/kubernetes.io~projected/kube-api-access-6xnhr DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c80c5b-5844-40f9-a886-13b5c312eb44/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df0d2140c2f779acefe3c8e1b3027f2030f52addd50a29e72dd4b8eacae0717b/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/038206a5-8ba4-456e-8e71-db0ec9cdd46c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7858d6a6-5909-4b5a-a5e1-fe41729234ec/volumes/kubernetes.io~projected/kube-api-access-x9j8j DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c243550e-f884-41f5-986f-69456856c89f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db4e1dcf-0e87-4463-a338-10fb9cf47d1a/volumes/kubernetes.io~projected/kube-api-access-kgbgj DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/dfab6970-50da-4943-9b93-f0f4089aacb2/volumes/kubernetes.io~projected/kube-api-access-9ldj7 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10f547184161d166427b397a3e56e090e7cc961081562a42a3ebce3ecba04cc3/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/98011d8e-1c32-46d2-ba93-b0a8ff367bc4/volumes/kubernetes.io~projected/kube-api-access-rfp4m DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da/volumes/kubernetes.io~projected/kube-api-access-shrwh DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~projected/kube-api-access-zdnv7 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa5e38f489447288a937be94a40ef03d983865a08fe63b926e5eb605be603213/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e0a8bf67afb3068a9e0deaf52f864db12a40bc87c7302542c025f428d2a04c5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/491877c9-e16e-405a-b84b-95db063f5b93/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e04297687efc0fffec1066967f121315f79ed7d66eaefdcd09726b14cad67d3/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45c07b65ef6350fa1329b6822376e8af18abc3020349cf1786174371f0ff4df5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46f06038-b436-45b8-9502-b9e2a4ddd4bf/volumes/kubernetes.io~projected/kube-api-access-fc4wh DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab5fe13c79a02a180b6f3f818311e14ae7b7cf768d5efe1d53e3a006db6c201c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65cc0067d080e010338f1c326490db3c79e9f1a0f15bebe4316d2c4a2e3bddf4/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a1aff89a-851d-4a9a-9b60-63c75683b36a/volumes/kubernetes.io~projected/kube-api-access-crsfv DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10f547184161d16 MacAddress:a2:b7:b5:a5:a4:05 Speed:10000 Mtu:8901} {Name:65cc0067d080e01 MacAddress:36:92:f8:3d:2e:9a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cb:35:82:db Speed:0 Mtu:9001} {Name:br-int MacAddress:86:58:46:42:ac:79 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:cb:35:82:db Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:46:42:44:59:37:f1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:34:8b:77:46:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162179    4274 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162234    4274 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162528    4274 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162551    4274 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-118.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162949    4274 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162961    4274 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162975    4274 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.162994    4274 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163116    4274 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163382    4274 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163569    4274 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163611    4274 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:29:35.167396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163625    4274 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163640    4274 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163656    4274 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.163668    4274 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.164631    4274 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.164646    4274 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.165486    4274 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.167492    4274 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.167502    4274 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:29:35.167935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.167603    4274 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:29:35.168404 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168384    4274 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:29:35.168503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168418    4274 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:29:35.168542 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168521    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:29:35.168542 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168534    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:29:35.168542 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168541    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168546    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168552    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168558    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168564    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168570    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168576    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168582    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168590    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168599    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168622    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:29:35.168653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.168627    4274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:29:35.171789 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.171777    4274 server.go:1263] "Started kubelet"
Nov 03 20:29:35.171924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.171874    4274 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:29:35.172052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.171893    4274 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:29:35.172137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.172084    4274 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:29:35.172392 ip-10-0-128-118 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:29:35.173217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.172640    4274 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:29:35.173893 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.173875    4274 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.174951    4274 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.174967    4274 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.175004    4274 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:55 +0000 UTC" deadline="2028-06-06 07:11:51.782379425 +0000 UTC"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.175066    4274 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13930h42m16.607316226s"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.175102    4274 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.175109    4274 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.175152    4274 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:29:35.176663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.175829    4274 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:29:35.177164 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177143    4274 factory.go:55] Registering systemd factory
Nov 03 20:29:35.177663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177167    4274 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:29:35.177663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177592    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7baebc9-c7a8-4ee7-8983-4721a97edef4" volumeName="kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:29:35.177663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177633    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfab6970-50da-4943-9b93-f0f4089aacb2" volumeName="kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:29:35.177663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177649    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f06038-b436-45b8-9502-b9e2a4ddd4bf" volumeName="kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:29:35.177663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177663    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c243550e-f884-41f5-986f-69456856c89f" volumeName="kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177675    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c243550e-f884-41f5-986f-69456856c89f" volumeName="kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177701    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfab6970-50da-4943-9b93-f0f4089aacb2" volumeName="kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177714    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c80c5b-5844-40f9-a886-13b5c312eb44" volumeName="kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177725    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da" volumeName="kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177740    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da" volumeName="kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177753    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f06038-b436-45b8-9502-b9e2a4ddd4bf" volumeName="kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177767    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491877c9-e16e-405a-b84b-95db063f5b93" volumeName="kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177779    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491877c9-e16e-405a-b84b-95db063f5b93" volumeName="kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177790    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfab6970-50da-4943-9b93-f0f4089aacb2" volumeName="kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177801    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7baebc9-c7a8-4ee7-8983-4721a97edef4" volumeName="kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177817    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7baebc9-c7a8-4ee7-8983-4721a97edef4" volumeName="kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177830    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c80c5b-5844-40f9-a886-13b5c312eb44" volumeName="kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177843    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c80c5b-5844-40f9-a886-13b5c312eb44" volumeName="kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177855    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038206a5-8ba4-456e-8e71-db0ec9cdd46c" volumeName="kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177869    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="491877c9-e16e-405a-b84b-95db063f5b93" volumeName="kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177884    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7858d6a6-5909-4b5a-a5e1-fe41729234ec" volumeName="kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177898    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98011d8e-1c32-46d2-ba93-b0a8ff367bc4" volumeName="kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177911    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1aff89a-851d-4a9a-9b60-63c75683b36a" volumeName="kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177922    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7baebc9-c7a8-4ee7-8983-4721a97edef4" volumeName="kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177933    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c80c5b-5844-40f9-a886-13b5c312eb44" volumeName="kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177945    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c80c5b-5844-40f9-a886-13b5c312eb44" volumeName="kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177962    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1aff89a-851d-4a9a-9b60-63c75683b36a" volumeName="kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177976    4274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db4e1dcf-0e87-4463-a338-10fb9cf47d1a" volumeName="kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj" seLinuxMountContext=""
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177990    4274 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:29:35.178003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.177998    4274 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178314    4274 factory.go:153] Registering CRI-O factory
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178332    4274 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178403    4274 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178424    4274 factory.go:103] Registering Raw factory
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178440    4274 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:29:35.179206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.178978    4274 manager.go:319] Starting recovery of all containers
Nov 03 20:29:35.203950 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.203906    4274 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:29:35.205668 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.205637    4274 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:29:35.205668 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.205668    4274 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:29:35.205841 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.205693    4274 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:29:35.205841 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.205737    4274 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:29:35.218087 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218052    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="8cb034d263b4210372f538677dfaa6d2f17d801eb38045ed8351bf8e2f1744d9" exitCode=0
Nov 03 20:29:35.218087 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218084    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="907640b104195607736ad7a683e9187666d5735af25892a9b7987c2ebf262602" exitCode=0
Nov 03 20:29:35.218250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218098    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="83116649adc7234057a2d2d7d35ebae7009b3b0b7fa42fcc12dc0b9dc213be85" exitCode=0
Nov 03 20:29:35.218250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218108    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="b74ee7447d48329b3d29755f3c4085f1961a17bf490df92de0bc9325b1dae511" exitCode=0
Nov 03 20:29:35.218250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218117    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="b03e30505cb8cf86a2ec415eb1d5f84ee7de674940dcda1e48b6f388b83a5c3c" exitCode=0
Nov 03 20:29:35.218250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.218125    4274 generic.go:356] "Generic (PLEG): container finished" podID="f7baebc9-c7a8-4ee7-8983-4721a97edef4" containerID="82a586279214e3909d9ed8efbcda16697009d44c6d980efbe7e84b326a7b79d5" exitCode=0
Nov 03 20:29:35.226459 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.226422    4274 generic.go:356] "Generic (PLEG): container finished" podID="8892d1609e982d25dd2b1fad0fa8e7c5" containerID="861165926a981577c38b058e20b369140bb15e3704e51393e2a64236c98c26f2" exitCode=0
Nov 03 20:29:35.239383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.239364    4274 manager.go:324] Recovery completed
Nov 03 20:29:35.250124 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.250101    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.251381 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.251364    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.251462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.251394    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.251462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.251405    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.253441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253420    4274 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:29:35.253524 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253455    4274 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:29:35.253524 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253475    4274 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:29:35.253687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253670    4274 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:29:35.253732 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253690    4274 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:29:35.253732 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253726    4274 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:29:35.253822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253735    4274 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:29:35.253822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253743    4274 policy_none.go:50] "Start"
Nov 03 20:29:35.253822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253762    4274 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:29:35.253822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253774    4274 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:29:35.254654 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253976    4274 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:29:35.254654 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.253994    4274 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:29:35.254654 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.254002    4274 policy_none.go:44] "Start"
Nov 03 20:29:35.254824 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.254688    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.255679 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.255651    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.255771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.255696    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.255771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.255713    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.258745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.258715    4274 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:29:35.261966 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.261943    4274 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:29:35.261966 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.261966    4274 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:29:35.262089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.261975    4274 server.go:87] "Starting device plugin registration server"
Nov 03 20:29:35.262270 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262257    4274 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:29:35.262331 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262269    4274 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:29:35.262385 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262353    4274 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:29:35.262484 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262451    4274 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:29:35.262592 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262493    4274 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:29:35.262592 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262499    4274 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:29:35.262689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.262567    4274 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:29:35.262480786Z","UUID":"1859a5ec-d9b5-4095-a9c9-508346c3dd0d","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:29:35.263742 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.263725    4274 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:29:35.263820 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.263763    4274 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-118.us-west-2.compute.internal\" not found"
Nov 03 20:29:35.264161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.264142    4274 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:29:35.264161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.264164    4274 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:29:35.307075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.306516    4274 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"]
Nov 03 20:29:35.307196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.307113    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.308250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.308202    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.308335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.308264    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.308335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.308277    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.308505 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.308495    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.308665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.308651    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.309447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309431    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.309538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309454    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.309538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309463    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.309538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309436    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.309538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309531    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.309718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309549    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.309718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309659    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerStarted","Data":"83ebdb50ccd90dbc8e216b2be5c3596634974e2143e12f3f88bbe007325fd590"}
Nov 03 20:29:35.309718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309693    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerDied","Data":"861165926a981577c38b058e20b369140bb15e3704e51393e2a64236c98c26f2"}
Nov 03 20:29:35.309718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309710    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal" event={"ID":"8892d1609e982d25dd2b1fad0fa8e7c5","Type":"ContainerStarted","Data":"70be9108803610b1575f4e03100c1dcb25b8d2bbaa6a139985d587f1440539ce"}
Nov 03 20:29:35.309877 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.309729    4274 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.309877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.309789    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.310661 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.310644    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.310733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.310669    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.310733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.310679    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.310797 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.310789    4274 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-118.us-west-2.compute.internal\" not found" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.362949 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.362907    4274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:29:35.363681 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.363653    4274 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:35.364224 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364202    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:29:35.364341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364252    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:29:35.364341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364270    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:29:35.364341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364294    4274 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.364474 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364352    4274 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:35.364670 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.364656    4274 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:35.376338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.376317    4274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.410965 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.410934    4274 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:29:35.455420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.455390    4274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:35.455557 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.455439    4274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.455557 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.455453    4274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.455557 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.455492    4274 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:29:35.503461 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.503431    4274 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-118.us-west-2.compute.internal" is owned by different node (oldNodeID="5b72223a-8db5-40c2-b382-df2eef7a044e", newNodeID=""), cleaning up...
Nov 03 20:29:35.503955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.503940    4274 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.504012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.504003    4274 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.504569 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.504555    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeNotReady"
Nov 03 20:29:35.504623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.504583    4274 setters.go:546] "Node became not ready" node="ip-10-0-128-118.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:29:35Z","lastTransitionTime":"2026-11-03T20:29:35Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 20:29:35.533544 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.533516    4274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:35.533679 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:35.533573    4274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.555806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.555784    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.555882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.555815    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.555882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.555838    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656433    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656464    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656481    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656526    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c4b446c071455ed0e37e47646678b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"1c4b446c071455ed0e37e47646678b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656550    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.656712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.656550    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8892d1609e982d25dd2b1fad0fa8e7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" (UID: \"8892d1609e982d25dd2b1fad0fa8e7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:35.770709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.770683    4274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-118.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:29:35.770868 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:35.770760    4274 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:29:36.165174 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.165145    4274 apiserver.go:52] "Watching apiserver"
Nov 03 20:29:36.240462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.240439    4274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:36.261046 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.261016    4274 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:29:36.261375 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.261357    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m7s5v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz","openshift-cluster-node-tuning-operator/tuned-6m9cr","openshift-dns/node-resolver-c6pqw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-n42rq","kube-system/konnectivity-agent-rqnkl","openshift-image-registry/node-ca-njl4k","openshift-multus/multus-skrbt","openshift-network-diagnostics/network-check-target-4gmmf","kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-8bqft","openshift-multus/multus-additional-cni-plugins-l44nz","openshift-network-operator/iptables-alerter-t5zmh"]
Nov 03 20:29:36.264897 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.264879    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.276822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.276798    4274 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:29:36.297157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.297139    4274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:29:36.297217 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:36.297185    4274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-118.us-west-2.compute.internal"
Nov 03 20:29:36.311209 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.311187    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\""
Nov 03 20:29:36.311305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.311286    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\""
Nov 03 20:29:36.311343 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.311290    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\""
Nov 03 20:29:36.311343 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.311311    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\""
Nov 03 20:29:36.311741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.311706    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:29:36.315053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.315008    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:29:36.315161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.315050    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.315161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.315111    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.315161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.315142    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\""
Nov 03 20:29:36.315303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.315180    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:29:36.360690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360664    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.360790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360693    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.360790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360709    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.360790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360725    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.360790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360741    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.360982 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360793    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.360982 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360847    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.360982 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360930    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.360982 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360970    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.360998    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsfv\" (UniqueName: \"kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361014    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361028    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361057    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361083    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361119    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp4m\" (UniqueName: \"kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.361186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361136    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361179    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9j8j\" (UniqueName: \"kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361223    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361288    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361337    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361366    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361391    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361408    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-tmp\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.361490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361482    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361502    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361536    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361555    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361590    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361612    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361628    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:36.361729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361643    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361767    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361903    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkfl\" (UniqueName: \"kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361921    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361927    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c243550e-f884-41f5-986f-69456856c89f-agent-certs\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361941    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361971    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.361992    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362007    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.362026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362024    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362062    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362082    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362099    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362118    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362135    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362202    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362251    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1aff89a-851d-4a9a-9b60-63c75683b36a-tmp-dir\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362253    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrwh\" (UniqueName: \"kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362291    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362382    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362408    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.362470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362434    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362481    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362503    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362525    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362571    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362603    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362599    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-serviceca\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362644    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnhr\" (UniqueName: \"kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362665    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362751    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362766    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362814    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362850    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.362915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362876    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362929    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362945    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362960    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362975    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.362992    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4wh\" (UniqueName: \"kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363022    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363070    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldj7\" (UniqueName: \"kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363088    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363109    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363126    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363165    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363190    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363207    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/491877c9-e16e-405a-b84b-95db063f5b93-etc-tuned\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363248    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363320    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363349    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c243550e-f884-41f5-986f-69456856c89f-konnectivity-ca\") pod \"konnectivity-agent-rqnkl\" (UID: \"c243550e-f884-41f5-986f-69456856c89f\") " pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363412    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363446    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363465    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.363527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363510    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnv7\" (UniqueName: \"kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.364114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363544    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.364114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.363648    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.411091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411065    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:29:36.411172 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411159    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:29:36.411212 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411171    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\""
Nov 03 20:29:36.411251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411223    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:36.411251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411246    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\""
Nov 03 20:29:36.411318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411265    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:36.411318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411274    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\""
Nov 03 20:29:36.411318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411269    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\""
Nov 03 20:29:36.411318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411296    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\""
Nov 03 20:29:36.411441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411333    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\""
Nov 03 20:29:36.411441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411424    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\""
Nov 03 20:29:36.411545 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411529    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovn-node-metrics-cert\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.411891 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411874    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:29:36.411891 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411887    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:29:36.411980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.411968    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:29:36.412082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412067    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:29:36.412138 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412116    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412138 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412133    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412243 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412213    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.412300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412252    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:29:36.412300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412264    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412271    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:29:36.412400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412341    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412386    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.412483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412461    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412482    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.412533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412481    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/038206a5-8ba4-456e-8e71-db0ec9cdd46c-original-pull-secret\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.412533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412525    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.412669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412550    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:29:36.412669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412570    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:29:36.412669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412594    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-binary-copy\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.412669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.412606    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.413238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.413218    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-env-overrides\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.413282 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.413244    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-config\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.413996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.413981    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-cni-binary-copy\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.415928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.415878    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsfv\" (UniqueName: \"kubernetes.io/projected/a1aff89a-851d-4a9a-9b60-63c75683b36a-kube-api-access-crsfv\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.416050 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.415989    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9j8j\" (UniqueName: \"kubernetes.io/projected/7858d6a6-5909-4b5a-a5e1-fe41729234ec-kube-api-access-x9j8j\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.416094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.416030    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp4m\" (UniqueName: \"kubernetes.io/projected/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kube-api-access-rfp4m\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.417549 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.417532    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnhr\" (UniqueName: \"kubernetes.io/projected/f7baebc9-c7a8-4ee7-8983-4721a97edef4-kube-api-access-6xnhr\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.417654 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.417639    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkfl\" (UniqueName: \"kubernetes.io/projected/f7c80c5b-5844-40f9-a886-13b5c312eb44-kube-api-access-4mkfl\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.421684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.421666    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c80c5b-5844-40f9-a886-13b5c312eb44-ovnkube-script-lib\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.421750 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.421668    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.421750 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.421703    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7baebc9-c7a8-4ee7-8983-4721a97edef4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.421817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.421779    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-daemon-config\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.422951 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.422934    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f06038-b436-45b8-9502-b9e2a4ddd4bf-iptables-alerter-script\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.457101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457081    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:29:36.457182 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457126    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.457182 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457133    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.457182 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457160    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:36.457508 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457481    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnv7\" (UniqueName: \"kubernetes.io/projected/491877c9-e16e-405a-b84b-95db063f5b93-kube-api-access-zdnv7\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.457585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457565    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldj7\" (UniqueName: \"kubernetes.io/projected/dfab6970-50da-4943-9b93-f0f4089aacb2-kube-api-access-9ldj7\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.457666 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.457654    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4wh\" (UniqueName: \"kubernetes.io/projected/46f06038-b436-45b8-9502-b9e2a4ddd4bf-kube-api-access-fc4wh\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.464361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464343    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464366    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464381    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.464418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464395    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.464418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464410    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464447    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-multus-certs\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464456    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-systemd\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464469    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-systemd-units\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464474    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464495    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-netns\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464510    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464514    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-modprobe-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464507    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-registration-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464542    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-sys\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464546    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.464594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464574    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464607    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-systemd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464623    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-ovn\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464627    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464644    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-k8s-cni-cncf-io\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464659    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464686    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464708    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464731    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-run-netns\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464732    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464771    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464769    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-slash\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464792    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464809    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-cnibin\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464810    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-socket-dir-parent\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464773    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1aff89a-851d-4a9a-9b60-63c75683b36a-hosts-file\") pod \"node-resolver-c6pqw\" (UID: \"a1aff89a-851d-4a9a-9b60-63c75683b36a\") " pod="openshift-dns/node-resolver-c6pqw"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464829    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464833    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-conf\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464854    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-log-socket\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464854    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464880    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464903    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464918    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464933    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464936    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-device-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464949    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464973    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464989    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-run\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464996    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.464900    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465006    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465015    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-etc-kubernetes\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465026    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465054    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-bin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465001    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-cni-multus\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465085    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-dbus\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465028    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysconfig\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465070    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465127    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-var-lib-kubelet\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465139    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465156    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465170    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465184    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465201    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465220    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-cnibin\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.465195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465219    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465238    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-host-var-lib-kubelet\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465249    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-sys-fs\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465254    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-netd\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465268    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465243    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-socket-dir\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465280    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-cni-bin\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465290    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465305    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465326    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465341    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465354    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465365    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-node-log\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465377    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98011d8e-1c32-46d2-ba93-b0a8ff367bc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-r28dz\" (UID: \"98011d8e-1c32-46d2-ba93-b0a8ff367bc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r28dz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465370    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465393    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f06038-b436-45b8-9502-b9e2a4ddd4bf-host-slash\") pod \"iptables-alerter-t5zmh\" (UID: \"46f06038-b436-45b8-9502-b9e2a4ddd4bf\") " pod="openshift-network-operator/iptables-alerter-t5zmh"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465420    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465440    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-etc-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465422    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-system-cni-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465462    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465475    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-sysctl-d\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465484    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-host\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465490    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465518    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465541    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-lib-modules\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465543    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465503    4274 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-multus-conf-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465551    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-run-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465585    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/038206a5-8ba4-456e-8e71-db0ec9cdd46c-kubelet-config\") pod \"global-pull-secret-syncer-m7s5v\" (UID: \"038206a5-8ba4-456e-8e71-db0ec9cdd46c\") " pod="kube-system/global-pull-secret-syncer-m7s5v"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465616    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465660    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465670    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7baebc9-c7a8-4ee7-8983-4721a97edef4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l44nz\" (UID: \"f7baebc9-c7a8-4ee7-8983-4721a97edef4\") " pod="openshift-multus/multus-additional-cni-plugins-l44nz"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465678    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465708    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465714    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-host\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465728    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465732    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/491877c9-e16e-405a-b84b-95db063f5b93-etc-kubernetes\") pod \"tuned-6m9cr\" (UID: \"491877c9-e16e-405a-b84b-95db063f5b93\") " pod="openshift-cluster-node-tuning-operator/tuned-6m9cr"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465755    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465760    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465756    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465786    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465801    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465810    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-host-kubelet\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465815    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465834    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-hostroot\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465883    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c80c5b-5844-40f9-a886-13b5c312eb44-var-lib-openvswitch\") pod \"ovnkube-node-n42rq\" (UID: \"f7c80c5b-5844-40f9-a886-13b5c312eb44\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.466773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.465928    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfab6970-50da-4943-9b93-f0f4089aacb2-system-cni-dir\") pod \"multus-skrbt\" (UID: \"dfab6970-50da-4943-9b93-f0f4089aacb2\") " pod="openshift-multus/multus-skrbt"
Nov 03 20:29:36.468129 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.467204    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrwh\" (UniqueName: \"kubernetes.io/projected/fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da-kube-api-access-shrwh\") pod \"node-ca-njl4k\" (UID: \"fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da\") " pod="openshift-image-registry/node-ca-njl4k"
Nov 03 20:29:36.468129 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.467428    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbgj\" (UniqueName: \"kubernetes.io/projected/db4e1dcf-0e87-4463-a338-10fb9cf47d1a-kube-api-access-kgbgj\") pod \"network-check-target-4gmmf\" (UID: \"db4e1dcf-0e87-4463-a338-10fb9cf47d1a\") " pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:36.468129 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.467569    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7858d6a6-5909-4b5a-a5e1-fe41729234ec-metrics-certs\") pod \"network-metrics-daemon-8bqft\" (UID: \"7858d6a6-5909-4b5a-a5e1-fe41729234ec\") " pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.565288 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.565263    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8bqft"
Nov 03 20:29:36.725972 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.725906    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:36.955590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.955552    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8bqft"]
Nov 03 20:29:36.959746 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:36.959717    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7858d6a6_5909_4b5a_a5e1_fe41729234ec.slice/crio-4961c6abb45a3914f50e7be9df7669ebe7477dca3bb32e044847b60969cd66e4 WatchSource:0}: Error finding container 4961c6abb45a3914f50e7be9df7669ebe7477dca3bb32e044847b60969cd66e4: Status 404 returned error can't find the container with id 4961c6abb45a3914f50e7be9df7669ebe7477dca3bb32e044847b60969cd66e4
Nov 03 20:29:36.961734 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:36.961718    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:29:37.123653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:37.123629    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:37.242866 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:37.242829    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bqft" event={"ID":"7858d6a6-5909-4b5a-a5e1-fe41729234ec","Type":"ContainerStarted","Data":"4961c6abb45a3914f50e7be9df7669ebe7477dca3bb32e044847b60969cd66e4"}
Nov 03 20:29:38.245468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:38.245443    4274 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:29:39.249693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:39.249658    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bqft" event={"ID":"7858d6a6-5909-4b5a-a5e1-fe41729234ec","Type":"ContainerStarted","Data":"14209b7d87f13c903324365b1352af3d22f4765ae35e6826e90039e0ad3995e3"}
Nov 03 20:29:39.250141 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:39.249708    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8bqft" event={"ID":"7858d6a6-5909-4b5a-a5e1-fe41729234ec","Type":"ContainerStarted","Data":"7e31a55dc01914bb88da4471153542817c0d05086607d94dba5cb817c41f0df5"}
Nov 03 20:29:41.544461 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:41.544429    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42rq"
Nov 03 20:29:42.895541 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.895513    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-n6k5m"]
Nov 03 20:29:42.898399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.898378    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w88bv"]
Nov 03 20:29:42.898527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.898494    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:42.901269 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.901250    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:42.941992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.941966    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-665c494c89-mpwvq"]
Nov 03 20:29:42.944524 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.944509    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:42.949930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.949910    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:29:42.949930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.949914    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:29:42.950107 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.949913    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:29:42.976349 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.976331    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qhrzx\""
Nov 03 20:29:42.976450 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.976360    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9t9mk\""
Nov 03 20:29:42.976450 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.976392    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:29:42.987936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.987911    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pmkxf"]
Nov 03 20:29:42.992506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.992484    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4mr9p"]
Nov 03 20:29:42.992700 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.992667    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:42.995267 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.995248    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:29:42.995507 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.995489    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:42.999112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.998983    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7faf2c9-dcf5-4430-9741-80055ccfc245-crio-socket\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:42.999112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999013    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7faf2c9-dcf5-4430-9741-80055ccfc245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:42.999112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999054    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d788f8d-574b-4604-b1b2-045ce07022f1-tmp\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:42.999274 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999188    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7faf2c9-dcf5-4430-9741-80055ccfc245-data-volume\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:42.999319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999281    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:42.999319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999310    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbht\" (UniqueName: \"kubernetes.io/projected/3d788f8d-574b-4604-b1b2-045ce07022f1-kube-api-access-vlbht\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:42.999409 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:42.999345    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt8l\" (UniqueName: \"kubernetes.io/projected/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-api-access-xtt8l\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.042516 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.042489    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 20:29:43.043222 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.043192    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:29:43.043222 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.043200    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n6k5m"]
Nov 03 20:29:43.043675 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.043648    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w88bv"]
Nov 03 20:29:43.046726 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.046709    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4mr9p"]
Nov 03 20:29:43.052933 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.052913    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pmkxf"]
Nov 03 20:29:43.068930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.068906    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 20:29:43.069324 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.069308    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 20:29:43.069417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.069348    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rrsvs\""
Nov 03 20:29:43.069417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.069349    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdt7f\""
Nov 03 20:29:43.070152 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.070135    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:29:43.070236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.070152    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:29:43.070236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.070135    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:29:43.070236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.070154    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 20:29:43.070236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.070190    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:29:43.079932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.079915    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:29:43.079992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.079920    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9nsh4\""
Nov 03 20:29:43.080587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.080573    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-665c494c89-mpwvq"]
Nov 03 20:29:43.100463 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100441    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f595b4e-61b2-442e-b8e8-3790ada9214a-ca-trust-extracted\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.100559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100473    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-trusted-ca\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.100559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100493    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53f3aef-556c-4bd8-ad54-2ddd522338b4-cert\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.100559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100514    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100571    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbht\" (UniqueName: \"kubernetes.io/projected/3d788f8d-574b-4604-b1b2-045ce07022f1-kube-api-access-vlbht\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100640    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2664af0a-7726-48b6-83df-96197730d5dc-metrics-tls\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100688    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fcd\" (UniqueName: \"kubernetes.io/projected/2664af0a-7726-48b6-83df-96197730d5dc-kube-api-access-n4fcd\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100744    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2664af0a-7726-48b6-83df-96197730d5dc-tmp-dir\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100817    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt8l\" (UniqueName: \"kubernetes.io/projected/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-api-access-xtt8l\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100845    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-certificates\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100862    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4cln\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-kube-api-access-r4cln\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.100877 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100880    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-installation-pull-secrets\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100905    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7faf2c9-dcf5-4430-9741-80055ccfc245-crio-socket\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100954    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7faf2c9-dcf5-4430-9741-80055ccfc245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100987    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7faf2c9-dcf5-4430-9741-80055ccfc245-crio-socket\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.100991    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2664af0a-7726-48b6-83df-96197730d5dc-config-volume\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101029    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-tls\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101109    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101129    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-bound-sa-token\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101173    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d788f8d-574b-4604-b1b2-045ce07022f1-tmp\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101200    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-image-registry-private-configuration\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.101236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101233    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7faf2c9-dcf5-4430-9741-80055ccfc245-data-volume\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.101538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101261    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bk6\" (UniqueName: \"kubernetes.io/projected/c53f3aef-556c-4bd8-ad54-2ddd522338b4-kube-api-access-l5bk6\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.101575 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101547    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d788f8d-574b-4604-b1b2-045ce07022f1-tmp\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:43.101575 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.101552    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7faf2c9-dcf5-4430-9741-80055ccfc245-data-volume\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.103203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.103181    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7faf2c9-dcf5-4430-9741-80055ccfc245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.157438 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.157388    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt8l\" (UniqueName: \"kubernetes.io/projected/d7faf2c9-dcf5-4430-9741-80055ccfc245-kube-api-access-xtt8l\") pod \"insights-runtime-extractor-w88bv\" (UID: \"d7faf2c9-dcf5-4430-9741-80055ccfc245\") " pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.157525 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.157470    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbht\" (UniqueName: \"kubernetes.io/projected/3d788f8d-574b-4604-b1b2-045ce07022f1-kube-api-access-vlbht\") pod \"downloads-6dd789c8b7-n6k5m\" (UID: \"3d788f8d-574b-4604-b1b2-045ce07022f1\") " pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:43.202276 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202251    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-image-registry-private-configuration\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202284    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bk6\" (UniqueName: \"kubernetes.io/projected/c53f3aef-556c-4bd8-ad54-2ddd522338b4-kube-api-access-l5bk6\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.202386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202300    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f595b4e-61b2-442e-b8e8-3790ada9214a-ca-trust-extracted\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202321    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-trusted-ca\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202352    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53f3aef-556c-4bd8-ad54-2ddd522338b4-cert\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.202593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202481    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2664af0a-7726-48b6-83df-96197730d5dc-metrics-tls\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.202593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202522    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fcd\" (UniqueName: \"kubernetes.io/projected/2664af0a-7726-48b6-83df-96197730d5dc-kube-api-access-n4fcd\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.202593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202552    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2664af0a-7726-48b6-83df-96197730d5dc-tmp-dir\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.202593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202585    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-certificates\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202611    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cln\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-kube-api-access-r4cln\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202640    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-installation-pull-secrets\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202683    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2664af0a-7726-48b6-83df-96197730d5dc-config-volume\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202686    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f595b4e-61b2-442e-b8e8-3790ada9214a-ca-trust-extracted\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202706    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-tls\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.202798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.202731    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-bound-sa-token\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.203604 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.203223    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2664af0a-7726-48b6-83df-96197730d5dc-tmp-dir\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.203604 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.203433    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2664af0a-7726-48b6-83df-96197730d5dc-config-volume\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.203604 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.203494    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-certificates\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.203604 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.203494    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f595b4e-61b2-442e-b8e8-3790ada9214a-trusted-ca\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.204872 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.204854    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-image-registry-private-configuration\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.205208 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.205190    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c53f3aef-556c-4bd8-ad54-2ddd522338b4-cert\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.205336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.205321    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2664af0a-7726-48b6-83df-96197730d5dc-metrics-tls\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.205380 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.205322    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f595b4e-61b2-442e-b8e8-3790ada9214a-installation-pull-secrets\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.205762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.205741    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-registry-tls\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.209703 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.209681    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:29:43.219088 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.219072    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w88bv"
Nov 03 20:29:43.262828 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.262765    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bk6\" (UniqueName: \"kubernetes.io/projected/c53f3aef-556c-4bd8-ad54-2ddd522338b4-kube-api-access-l5bk6\") pod \"ingress-canary-4mr9p\" (UID: \"c53f3aef-556c-4bd8-ad54-2ddd522338b4\") " pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.263823 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.263763    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-bound-sa-token\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.263933 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.263902    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fcd\" (UniqueName: \"kubernetes.io/projected/2664af0a-7726-48b6-83df-96197730d5dc-kube-api-access-n4fcd\") pod \"dns-default-pmkxf\" (UID: \"2664af0a-7726-48b6-83df-96197730d5dc\") " pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.266161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.266030    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cln\" (UniqueName: \"kubernetes.io/projected/5f595b4e-61b2-442e-b8e8-3790ada9214a-kube-api-access-r4cln\") pod \"image-registry-665c494c89-mpwvq\" (UID: \"5f595b4e-61b2-442e-b8e8-3790ada9214a\") " pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.307472 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.307447    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:43.312872 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.312786    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4mr9p"
Nov 03 20:29:43.554598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.554571    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:43.577876 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.577850    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n6k5m"]
Nov 03 20:29:43.581140 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:43.581112    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d788f8d_574b_4604_b1b2_045ce07022f1.slice/crio-196ebd7eac3e3e503285f63072dc914d775b5017bffeffb3f31c3c1486187941 WatchSource:0}: Error finding container 196ebd7eac3e3e503285f63072dc914d775b5017bffeffb3f31c3c1486187941: Status 404 returned error can't find the container with id 196ebd7eac3e3e503285f63072dc914d775b5017bffeffb3f31c3c1486187941
Nov 03 20:29:43.619091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.619067    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w88bv"]
Nov 03 20:29:43.625378 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:43.625349    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7faf2c9_dcf5_4430_9741_80055ccfc245.slice/crio-7d013ffb5e852eb82c6e779220e0d60f76236cb775f9683bbd7e775147a4ec72 WatchSource:0}: Error finding container 7d013ffb5e852eb82c6e779220e0d60f76236cb775f9683bbd7e775147a4ec72: Status 404 returned error can't find the container with id 7d013ffb5e852eb82c6e779220e0d60f76236cb775f9683bbd7e775147a4ec72
Nov 03 20:29:43.682908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.682886    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4mr9p"]
Nov 03 20:29:43.686081 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:43.686053    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53f3aef_556c_4bd8_ad54_2ddd522338b4.slice/crio-35cf47d92a47261fec5f58c5ce92296f104c5eb885ba345f034211effc027c37 WatchSource:0}: Error finding container 35cf47d92a47261fec5f58c5ce92296f104c5eb885ba345f034211effc027c37: Status 404 returned error can't find the container with id 35cf47d92a47261fec5f58c5ce92296f104c5eb885ba345f034211effc027c37
Nov 03 20:29:43.691078 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.691057    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pmkxf"]
Nov 03 20:29:43.693771 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:43.693747    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2664af0a_7726_48b6_83df_96197730d5dc.slice/crio-c75a0d012982b8e546768040279220c91140305a96f5d7a9281d33a27fd33391 WatchSource:0}: Error finding container c75a0d012982b8e546768040279220c91140305a96f5d7a9281d33a27fd33391: Status 404 returned error can't find the container with id c75a0d012982b8e546768040279220c91140305a96f5d7a9281d33a27fd33391
Nov 03 20:29:43.839223 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.839151    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:29:43.919558 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:43.919527    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-665c494c89-mpwvq"]
Nov 03 20:29:43.922451 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:43.922412    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f595b4e_61b2_442e_b8e8_3790ada9214a.slice/crio-df75bb11fcce0984d5ac10b6a85c2f7f6012450c668d6f24b8bd053fd8986494 WatchSource:0}: Error finding container df75bb11fcce0984d5ac10b6a85c2f7f6012450c668d6f24b8bd053fd8986494: Status 404 returned error can't find the container with id df75bb11fcce0984d5ac10b6a85c2f7f6012450c668d6f24b8bd053fd8986494
Nov 03 20:29:44.263899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.263860    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-665c494c89-mpwvq" event={"ID":"5f595b4e-61b2-442e-b8e8-3790ada9214a","Type":"ContainerStarted","Data":"995868a31c08d0e6b1fb130890b1be167af4564b07ee9f85018b652022671864"}
Nov 03 20:29:44.264093 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.263922    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-665c494c89-mpwvq" event={"ID":"5f595b4e-61b2-442e-b8e8-3790ada9214a","Type":"ContainerStarted","Data":"df75bb11fcce0984d5ac10b6a85c2f7f6012450c668d6f24b8bd053fd8986494"}
Nov 03 20:29:44.265253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.265224    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmkxf" event={"ID":"2664af0a-7726-48b6-83df-96197730d5dc","Type":"ContainerStarted","Data":"c75a0d012982b8e546768040279220c91140305a96f5d7a9281d33a27fd33391"}
Nov 03 20:29:44.266614 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.266590    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w88bv" event={"ID":"d7faf2c9-dcf5-4430-9741-80055ccfc245","Type":"ContainerStarted","Data":"482e89ed4240ed5d7b158582f8be56bea6927a81dc5ca12ab561b66fc9685d84"}
Nov 03 20:29:44.266711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.266635    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w88bv" event={"ID":"d7faf2c9-dcf5-4430-9741-80055ccfc245","Type":"ContainerStarted","Data":"7d013ffb5e852eb82c6e779220e0d60f76236cb775f9683bbd7e775147a4ec72"}
Nov 03 20:29:44.267580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.267559    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4mr9p" event={"ID":"c53f3aef-556c-4bd8-ad54-2ddd522338b4","Type":"ContainerStarted","Data":"35cf47d92a47261fec5f58c5ce92296f104c5eb885ba345f034211effc027c37"}
Nov 03 20:29:44.269439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.269377    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n6k5m" event={"ID":"3d788f8d-574b-4604-b1b2-045ce07022f1","Type":"ContainerStarted","Data":"196ebd7eac3e3e503285f63072dc914d775b5017bffeffb3f31c3c1486187941"}
Nov 03 20:29:44.377661 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:44.377086    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-665c494c89-mpwvq" podStartSLOduration=2.377072455 podStartE2EDuration="2.377072455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:44.376710986 +0000 UTC m=+9.303281215" watchObservedRunningTime="2026-11-03 20:29:44.377072455 +0000 UTC m=+9.303642693"
Nov 03 20:29:45.273976 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:45.273950    4274 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:29:45.492718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:45.492646    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:29:46.281507 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:46.281464    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w88bv" event={"ID":"d7faf2c9-dcf5-4430-9741-80055ccfc245","Type":"ContainerStarted","Data":"32deb03b37674fb5bae23c3012469f2522ea009c686264cd40df591cb53ae853"}
Nov 03 20:29:47.287516 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:47.287483    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmkxf" event={"ID":"2664af0a-7726-48b6-83df-96197730d5dc","Type":"ContainerStarted","Data":"5f35f6aca749c540834d03d9db5515c2b02cda1c519bdcc9973615eaf684b080"}
Nov 03 20:29:47.287897 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:47.287546    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pmkxf" event={"ID":"2664af0a-7726-48b6-83df-96197730d5dc","Type":"ContainerStarted","Data":"eab794cb12e6f80c6e5d39b09dbdb4b68ed5f079a694ed4990b4d6cbbeb61de0"}
Nov 03 20:29:47.287897 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:47.287659    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:47.401334 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:47.401277    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pmkxf" podStartSLOduration=2.579808028 podStartE2EDuration="5.401264108s" totalImagesPullingTime="2.82145608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:47.400627222 +0000 UTC m=+12.327197459" watchObservedRunningTime="2026-11-03 20:29:47.401264108 +0000 UTC m=+12.327834398"
Nov 03 20:29:48.292687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:48.292647    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w88bv" event={"ID":"d7faf2c9-dcf5-4430-9741-80055ccfc245","Type":"ContainerStarted","Data":"26ebe73635c5e358aa020faa54086d98aac603c8b97adcc65ad3fd514ce6c867"}
Nov 03 20:29:48.294721 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:48.294695    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4mr9p" event={"ID":"c53f3aef-556c-4bd8-ad54-2ddd522338b4","Type":"ContainerStarted","Data":"c48f0ef431d8b9b726798f1303de973b4fa67479c9c5935bd691ad47eeebae45"}
Nov 03 20:29:48.403050 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:48.402990    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w88bv" podStartSLOduration=2.587963695 podStartE2EDuration="6.40297215s" totalImagesPullingTime="3.815008455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:48.402455664 +0000 UTC m=+13.329025903" watchObservedRunningTime="2026-11-03 20:29:48.40297215 +0000 UTC m=+13.329542385"
Nov 03 20:29:48.518606 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:48.518552    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4mr9p" podStartSLOduration=2.5692285630000002 podStartE2EDuration="6.518535307s" totalImagesPullingTime="3.949306744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:48.51832875 +0000 UTC m=+13.444898988" watchObservedRunningTime="2026-11-03 20:29:48.518535307 +0000 UTC m=+13.445105542"
Nov 03 20:29:49.249928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.249894    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"]
Nov 03 20:29:49.253995 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.253969    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:49.348390 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.348360    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 20:29:49.348794 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.348537    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-75lvf\""
Nov 03 20:29:49.349424 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.349403    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"]
Nov 03 20:29:49.352966 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.352942    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/deb2e161-ec98-4f5d-9da4-c648984b9ae8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-89pt2\" (UID: \"deb2e161-ec98-4f5d-9da4-c648984b9ae8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:49.454118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.454088    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/deb2e161-ec98-4f5d-9da4-c648984b9ae8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-89pt2\" (UID: \"deb2e161-ec98-4f5d-9da4-c648984b9ae8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:49.456739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.456714    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/deb2e161-ec98-4f5d-9da4-c648984b9ae8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-89pt2\" (UID: \"deb2e161-ec98-4f5d-9da4-c648984b9ae8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:49.570752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.570719    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:49.950201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:49.950097    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"]
Nov 03 20:29:49.953706 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:29:49.953676    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb2e161_ec98_4f5d_9da4_c648984b9ae8.slice/crio-f53c5ba16b108603af42b4e9f85141bc121945708f78e61aca48e7143e4a0188 WatchSource:0}: Error finding container f53c5ba16b108603af42b4e9f85141bc121945708f78e61aca48e7143e4a0188: Status 404 returned error can't find the container with id f53c5ba16b108603af42b4e9f85141bc121945708f78e61aca48e7143e4a0188
Nov 03 20:29:50.302095 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:50.302055    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2" event={"ID":"deb2e161-ec98-4f5d-9da4-c648984b9ae8","Type":"ContainerStarted","Data":"f53c5ba16b108603af42b4e9f85141bc121945708f78e61aca48e7143e4a0188"}
Nov 03 20:29:51.672256 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:51.672236    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqnkl"
Nov 03 20:29:52.310075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:52.310021    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2" event={"ID":"deb2e161-ec98-4f5d-9da4-c648984b9ae8","Type":"ContainerStarted","Data":"140f810d849834ff060564c8d5993d25c6414ad3c6bfa256a74cd6815c975315"}
Nov 03 20:29:52.310297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:52.310279    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:52.315225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:52.315194    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2"
Nov 03 20:29:52.417832 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:52.417749    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-89pt2" podStartSLOduration=1.705441117 podStartE2EDuration="3.417737298s" totalImagesPullingTime="1.712296181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:52.417509762 +0000 UTC m=+17.344080001" watchObservedRunningTime="2026-11-03 20:29:52.417737298 +0000 UTC m=+17.344307558"
Nov 03 20:29:57.297393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:29:57.297361    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pmkxf"
Nov 03 20:29:57.368919 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:29:57.368880    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:30:00.630498 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.630414    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7qfl7"]
Nov 03 20:30:00.635343 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.635315    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.678171 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.678139    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"]
Nov 03 20:30:00.681698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.681669    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721557    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-root\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.721736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721596    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721625    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-metrics-client-ca\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.721855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721753    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db69f86-491b-451b-b4d1-957b106365dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721797    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721829    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz9j\" (UniqueName: \"kubernetes.io/projected/8db69f86-491b-451b-b4d1-957b106365dc-kube-api-access-qwz9j\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721876    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.721999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721904    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-sys\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.721999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721937    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.721999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721968    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-tls\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.721999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.721993    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-wtmp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.722268 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.722015    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.722268 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.722064    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stzp\" (UniqueName: \"kubernetes.io/projected/163948e9-a5fc-4339-bee1-0a409da473c8-kube-api-access-9stzp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.722268 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.722097    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.722268 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.722146    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-textfile\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.723298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.723278    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:30:00.723509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.723488    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4nsd9\""
Nov 03 20:30:00.723589 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.723533    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:30:00.724007 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.723988    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:30:00.739093 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.739073    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:30:00.739468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.739444    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:30:00.739547 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.739511    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:30:00.772516 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.772488    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"]
Nov 03 20:30:00.797815 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.797790    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 20:30:00.797981 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.797913    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nqqzv\""
Nov 03 20:30:00.797981 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.797931    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:30:00.798257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.798227    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 20:30:00.822709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822672    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db69f86-491b-451b-b4d1-957b106365dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.822882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822719    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.822882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822752    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz9j\" (UniqueName: \"kubernetes.io/projected/8db69f86-491b-451b-b4d1-957b106365dc-kube-api-access-qwz9j\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.822882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822794    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.822882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822823    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-sys\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.822882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.822853    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823171    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-tls\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823258 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823214    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-wtmp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823258 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823244    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823255    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8db69f86-491b-451b-b4d1-957b106365dc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.823366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823268    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stzp\" (UniqueName: \"kubernetes.io/projected/163948e9-a5fc-4339-bee1-0a409da473c8-kube-api-access-9stzp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823297    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.823366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823352    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-textfile\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823382    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-root\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823409    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.823571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823415    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-wtmp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823417    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823444    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-metrics-client-ca\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.823817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.823817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.823634    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-sys\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.824712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.824055    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163948e9-a5fc-4339-bee1-0a409da473c8-metrics-client-ca\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.824712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.824071    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.824712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.824327    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-textfile\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.824712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.824390    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/163948e9-a5fc-4339-bee1-0a409da473c8-root\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.826494 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.826468    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.826788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.826714    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/163948e9-a5fc-4339-bee1-0a409da473c8-node-exporter-tls\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.826981 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.826960    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.827510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.827482    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8db69f86-491b-451b-b4d1-957b106365dc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.881395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.881314    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz9j\" (UniqueName: \"kubernetes.io/projected/8db69f86-491b-451b-b4d1-957b106365dc-kube-api-access-qwz9j\") pod \"kube-state-metrics-5dc8d7d89b-bqp2m\" (UID: \"8db69f86-491b-451b-b4d1-957b106365dc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:00.882125 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.882098    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stzp\" (UniqueName: \"kubernetes.io/projected/163948e9-a5fc-4339-bee1-0a409da473c8-kube-api-access-9stzp\") pod \"node-exporter-7qfl7\" (UID: \"163948e9-a5fc-4339-bee1-0a409da473c8\") " pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.953973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.953939    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qfl7"
Nov 03 20:30:00.999000 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:00.998972    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"
Nov 03 20:30:01.350979 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.350931    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n6k5m" event={"ID":"3d788f8d-574b-4604-b1b2-045ce07022f1","Type":"ContainerStarted","Data":"7da3b366be82d7c3e7fbe5885c71b809eda3ccf44a34b5c5cc6fbe0156be7a3e"}
Nov 03 20:30:01.351207 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.351168    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:30:01.352737 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.352700    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qfl7" event={"ID":"163948e9-a5fc-4339-bee1-0a409da473c8","Type":"ContainerStarted","Data":"364b419e3b1f9c72939a910d874dcf5af37f28f009f37c5e8ee36b24780e100c"}
Nov 03 20:30:01.368597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.368572    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-n6k5m"
Nov 03 20:30:01.416817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.416738    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m"]
Nov 03 20:30:01.421238 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:30:01.421199    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db69f86_491b_451b_b4d1_957b106365dc.slice/crio-9f9bc7152bc0e3c911a73cb9c1fcd17d1db7cc3bb162eeb8c61e48abbee5817a WatchSource:0}: Error finding container 9f9bc7152bc0e3c911a73cb9c1fcd17d1db7cc3bb162eeb8c61e48abbee5817a: Status 404 returned error can't find the container with id 9f9bc7152bc0e3c911a73cb9c1fcd17d1db7cc3bb162eeb8c61e48abbee5817a
Nov 03 20:30:01.461735 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.461681    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-n6k5m" podStartSLOduration=2.699439619 podStartE2EDuration="19.461664134s" totalImagesPullingTime="16.762224515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.461494674 +0000 UTC m=+26.388064914" watchObservedRunningTime="2026-11-03 20:30:01.461664134 +0000 UTC m=+26.388234373"
Nov 03 20:30:01.854875 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.853417    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:01.880536 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.880505    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.915516 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.915480    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:01.931783 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.931752    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.931938 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.931802    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.931938 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.931845    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.931938 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.931893    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.932113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.931989    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.932113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.932015    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sf7\" (UniqueName: \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.932113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.932082    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.932257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.932162    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:01.934659 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.934592    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:30:01.934769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.934612    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:30:01.934769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.934636    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:30:01.981867 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.981829    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g5f8g\""
Nov 03 20:30:01.982053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.981868    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:30:01.982053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.981882    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:30:01.982203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:01.982186    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:30:02.033222 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033176    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033222 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033232    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033275    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033297    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033329    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033354    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sf7\" (UniqueName: \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.033908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033880    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.034049 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033938    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.034049 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.033998    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.034399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.034354    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.034926 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.034899    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.035015 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.034932    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.035217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.035196    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.036525 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.036500    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.036741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.036720    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.091658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.091570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sf7\" (UniqueName: \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\") pod \"console-7b56cc7857-zgtcl\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") " pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.213431 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.213351    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:02.359782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.359751    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m" event={"ID":"8db69f86-491b-451b-b4d1-957b106365dc","Type":"ContainerStarted","Data":"9f9bc7152bc0e3c911a73cb9c1fcd17d1db7cc3bb162eeb8c61e48abbee5817a"}
Nov 03 20:30:02.616128 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:02.616088    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:02.628626 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:30:02.628255    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8584e1f_2f35_4ee7_ba7f_2375e9007803.slice/crio-c824dd8539e8badfdebf74e9db7f9334dc614df349b13cd25e412fcef4f69442 WatchSource:0}: Error finding container c824dd8539e8badfdebf74e9db7f9334dc614df349b13cd25e412fcef4f69442: Status 404 returned error can't find the container with id c824dd8539e8badfdebf74e9db7f9334dc614df349b13cd25e412fcef4f69442
Nov 03 20:30:03.365299 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:03.365204    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56cc7857-zgtcl" event={"ID":"c8584e1f-2f35-4ee7-ba7f-2375e9007803","Type":"ContainerStarted","Data":"c824dd8539e8badfdebf74e9db7f9334dc614df349b13cd25e412fcef4f69442"}
Nov 03 20:30:03.367575 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:03.367542    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qfl7" event={"ID":"163948e9-a5fc-4339-bee1-0a409da473c8","Type":"ContainerStarted","Data":"ed055f89da5640ef9838c232b9a24070a750471c1b63cff0f7c84cf8c5b85661"}
Nov 03 20:30:03.842468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:03.842434    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gmmf"
Nov 03 20:30:04.382410 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:04.382376    4274 generic.go:356] "Generic (PLEG): container finished" podID="163948e9-a5fc-4339-bee1-0a409da473c8" containerID="ed055f89da5640ef9838c232b9a24070a750471c1b63cff0f7c84cf8c5b85661" exitCode=0
Nov 03 20:30:04.382840 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:04.382492    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qfl7" event={"ID":"163948e9-a5fc-4339-bee1-0a409da473c8","Type":"ContainerDied","Data":"ed055f89da5640ef9838c232b9a24070a750471c1b63cff0f7c84cf8c5b85661"}
Nov 03 20:30:04.391600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:04.391575    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m" event={"ID":"8db69f86-491b-451b-b4d1-957b106365dc","Type":"ContainerStarted","Data":"7a0da79e754b75a11ba843636442fdf79d5e166c834509c33af380f7bfbff3cc"}
Nov 03 20:30:04.391692 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:04.391627    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m" event={"ID":"8db69f86-491b-451b-b4d1-957b106365dc","Type":"ContainerStarted","Data":"5f94d7808bbe7188abbfb7aece857a64164ca5a5d575f4e751f826713dbff4ef"}
Nov 03 20:30:05.401061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:05.400957    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qfl7" event={"ID":"163948e9-a5fc-4339-bee1-0a409da473c8","Type":"ContainerStarted","Data":"9388d3d006473334caa71770c9ef3c22cd7f02126fbff849d5ecc9ef76311057"}
Nov 03 20:30:05.401061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:05.401045    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qfl7" event={"ID":"163948e9-a5fc-4339-bee1-0a409da473c8","Type":"ContainerStarted","Data":"7e0b2e9d7d8a623693486e89d9ec9777fd2bbf1a89aeeb5c9d3acfe24239544b"}
Nov 03 20:30:05.405771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:05.405741    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m" event={"ID":"8db69f86-491b-451b-b4d1-957b106365dc","Type":"ContainerStarted","Data":"4d93b51a99aedff675a7a87e8f87100ee9330da3894b177c4c67d691f6faf37e"}
Nov 03 20:30:05.517235 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:05.517169    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7qfl7" podStartSLOduration=3.173588053 podStartE2EDuration="5.517149414s" totalImagesPullingTime="2.258433787s" totalInitContainerRuntime="85.127574ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:05.516606204 +0000 UTC m=+30.443176444" watchObservedRunningTime="2026-11-03 20:30:05.517149414 +0000 UTC m=+30.443719651"
Nov 03 20:30:06.287180 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:06.287151    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-665c494c89-mpwvq"
Nov 03 20:30:06.444514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:06.444458    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bqp2m" podStartSLOduration=3.755434804 podStartE2EDuration="6.444441188s" totalImagesPullingTime="2.689006384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:05.629440054 +0000 UTC m=+30.556010293" watchObservedRunningTime="2026-11-03 20:30:06.444441188 +0000 UTC m=+31.371011426"
Nov 03 20:30:07.416559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:07.416529    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56cc7857-zgtcl" event={"ID":"c8584e1f-2f35-4ee7-ba7f-2375e9007803","Type":"ContainerStarted","Data":"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"}
Nov 03 20:30:08.531690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:08.531634    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b56cc7857-zgtcl" podStartSLOduration=2.860174685 podStartE2EDuration="7.531621215s" totalImagesPullingTime="4.67144653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:08.531123389 +0000 UTC m=+33.457693628" watchObservedRunningTime="2026-11-03 20:30:08.531621215 +0000 UTC m=+33.458191452"
Nov 03 20:30:11.141960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:11.141928    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:12.214745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:12.214716    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:27.363710 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:30:27.363674    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:30:36.161559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.161488    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b56cc7857-zgtcl" podUID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" containerName="console" containerID="cri-o://a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86" gracePeriod=15
Nov 03 20:30:36.442252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.442227    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:36.504790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.504762    4274 generic.go:356] "Generic (PLEG): container finished" podID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" containerID="a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86" exitCode=0
Nov 03 20:30:36.504922 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.504794    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56cc7857-zgtcl" event={"ID":"c8584e1f-2f35-4ee7-ba7f-2375e9007803","Type":"ContainerDied","Data":"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"}
Nov 03 20:30:36.504922 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.504809    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b56cc7857-zgtcl"
Nov 03 20:30:36.504922 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.504857    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b56cc7857-zgtcl" event={"ID":"c8584e1f-2f35-4ee7-ba7f-2375e9007803","Type":"ContainerDied","Data":"c824dd8539e8badfdebf74e9db7f9334dc614df349b13cd25e412fcef4f69442"}
Nov 03 20:30:36.504922 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.504876    4274 scope.go:122] "RemoveContainer" containerID="a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"
Nov 03 20:30:36.524528 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.524511    4274 scope.go:122] "RemoveContainer" containerID="a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"
Nov 03 20:30:36.524778 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:30:36.524754    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86\": container with ID starting with a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86 not found: ID does not exist" containerID="a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"
Nov 03 20:30:36.524876 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.524785    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86"} err="failed to get container status \"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86\": rpc error: code = NotFound desc = could not find container \"a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86\": container with ID starting with a0c1556b3b9cda45beb55fbcab9fc9bf37511dda50e5f878121c4255d01f2d86 not found: ID does not exist"
Nov 03 20:30:36.588687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588658    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588702    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588721    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\" (UniqueName: \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588742    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588773    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588983 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588824    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.588983 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588873    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.589108 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.588999    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\") pod \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\" (UID: \"c8584e1f-2f35-4ee7-ba7f-2375e9007803\") "
Nov 03 20:30:36.589108 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589075    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:36.589261 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589210    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:36.589261 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589224    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:36.589451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589269    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:36.589451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589321    4274 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-oauth-serving-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.589451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589342    4274 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.589451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589357    4274 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8584e1f-2f35-4ee7-ba7f-2375e9007803-tmp\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.589602 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.589546    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:36.591008 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.590988    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:36.591122 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.591103    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "kube-api-access-m4sf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:36.591204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.591191    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert" pod "c8584e1f-2f35-4ee7-ba7f-2375e9007803" (UID: "c8584e1f-2f35-4ee7-ba7f-2375e9007803"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:36.690590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.690532    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4sf7\" (UniqueName: \"kubernetes.io/projected/c8584e1f-2f35-4ee7-ba7f-2375e9007803-kube-api-access-m4sf7\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.690590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.690555    4274 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-oauth-config\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.690590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.690566    4274 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-config\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.690590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.690575    4274 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8584e1f-2f35-4ee7-ba7f-2375e9007803-console-serving-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.690590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.690584    4274 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8584e1f-2f35-4ee7-ba7f-2375e9007803-trusted-ca-bundle\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:36.916204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.916173    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:36.921970 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:36.921945    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b56cc7857-zgtcl"]
Nov 03 20:30:37.210315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:37.210284    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" path="/var/lib/kubelet/pods/c8584e1f-2f35-4ee7-ba7f-2375e9007803/volumes"
Nov 03 20:30:52.233012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.232981    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b9685c66-sdgxz"]
Nov 03 20:30:52.233529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.233235    4274 cpu_manager.go:414] "containerMap: removing container" podUID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" containerName="console"
Nov 03 20:30:52.233529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.233250    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" containerName="console"
Nov 03 20:30:52.233529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.233314    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8584e1f-2f35-4ee7-ba7f-2375e9007803" containerName="console"
Nov 03 20:30:52.255233 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.255212    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.292667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.292642    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b9685c66-sdgxz"]
Nov 03 20:30:52.337927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.337864    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 20:30:52.337927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.337865    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 20:30:52.337927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.337876    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 20:30:52.338174 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.337956    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 20:30:52.338174 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.338133    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nl5zj\""
Nov 03 20:30:52.396158 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.396140    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 20:30:52.396328 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.396315    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 20:30:52.397435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397415    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397463    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397551 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397534    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397604 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397563    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397662 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397642    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-serving-certs-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397714 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397680    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-federate-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397775 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397758    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7254\" (UniqueName: \"kubernetes.io/projected/2feccb21-77ca-40e0-975b-2c0492f4286f-kube-api-access-z7254\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.397820 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.397788    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-metrics-client-ca\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498475    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498511    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498528    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498566    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498604    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-serving-certs-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498627    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-federate-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498848 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498756    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7254\" (UniqueName: \"kubernetes.io/projected/2feccb21-77ca-40e0-975b-2c0492f4286f-kube-api-access-z7254\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.498848 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.498794    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-metrics-client-ca\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.499338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.499316    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-serving-certs-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.499424 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.499410    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-metrics-client-ca\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.499510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.499484    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.501075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.501017    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.501331 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.501309    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-federate-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.501409 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.501361    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-secret-telemeter-client\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.501465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.501425    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2feccb21-77ca-40e0-975b-2c0492f4286f-telemeter-client-tls\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.555830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.555803    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7254\" (UniqueName: \"kubernetes.io/projected/2feccb21-77ca-40e0-975b-2c0492f4286f-kube-api-access-z7254\") pod \"telemeter-client-b9685c66-sdgxz\" (UID: \"2feccb21-77ca-40e0-975b-2c0492f4286f\") " pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:52.567634 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:52.567607    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz"
Nov 03 20:30:53.024784 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:53.024750    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b9685c66-sdgxz"]
Nov 03 20:30:53.027050 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:30:53.027005    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2feccb21_77ca_40e0_975b_2c0492f4286f.slice/crio-cd6ea5de410ab9e5c7a18cfa3a5e90392b3cd034c1e8cc4f740f9dd4f3c66ec9 WatchSource:0}: Error finding container cd6ea5de410ab9e5c7a18cfa3a5e90392b3cd034c1e8cc4f740f9dd4f3c66ec9: Status 404 returned error can't find the container with id cd6ea5de410ab9e5c7a18cfa3a5e90392b3cd034c1e8cc4f740f9dd4f3c66ec9
Nov 03 20:30:53.560212 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:53.560175    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz" event={"ID":"2feccb21-77ca-40e0-975b-2c0492f4286f","Type":"ContainerStarted","Data":"cd6ea5de410ab9e5c7a18cfa3a5e90392b3cd034c1e8cc4f740f9dd4f3c66ec9"}
Nov 03 20:30:56.571940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:56.571910    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz" event={"ID":"2feccb21-77ca-40e0-975b-2c0492f4286f","Type":"ContainerStarted","Data":"191acbadf6e0aa4e7bd45650d32227903d08ed1956484216277cb350f090f935"}
Nov 03 20:30:57.365173 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:30:57.365130    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:30:58.580451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:58.580415    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz" event={"ID":"2feccb21-77ca-40e0-975b-2c0492f4286f","Type":"ContainerStarted","Data":"263c2003996fdc231d9a78c035ff32caf72f9bac25dd9319a43acab8f154e172"}
Nov 03 20:30:58.580827 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:58.580472    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz" event={"ID":"2feccb21-77ca-40e0-975b-2c0492f4286f","Type":"ContainerStarted","Data":"49b4943b2e3a85a5a628213b3437154894cf9cca7d052514760e42be530173dd"}
Nov 03 20:30:58.788001 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:58.787947    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b9685c66-sdgxz" podStartSLOduration=2.251548933 podStartE2EDuration="6.787934651s" totalImagesPullingTime="4.536385718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:58.787013351 +0000 UTC m=+83.713583614" watchObservedRunningTime="2026-11-03 20:30:58.787934651 +0000 UTC m=+83.714504888"
Nov 03 20:30:59.146977 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.146947    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:30:59.150082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.150066    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.199290 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.199260    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:30:59.199449 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.199298    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:30:59.199449 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.199319    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:30:59.199572 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.199498    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g5f8g\""
Nov 03 20:30:59.199572 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.199518    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:30:59.245688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.245669    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:30:59.251466 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251438    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxhq\" (UniqueName: \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251599 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251479    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251599 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251519    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251599 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251540    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251764 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251711    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251764 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251745    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251867 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251827    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.251867 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.251853    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.292429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.292392    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:30:59.293374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.293356    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:30:59.352881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.352855    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.352973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.352884    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.352973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.352908    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.352973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.352922    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353112    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxhq\" (UniqueName: \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353144    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353201    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353227    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353354 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353323    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353668    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353787 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353726    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.353914 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.353893    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.354142 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.354123    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.355591 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.355570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.355671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.355570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.410367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.410301    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxhq\" (UniqueName: \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\") pod \"console-f84d59654-mhsd5\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") " pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.460779 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.460747    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:30:59.822845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:30:59.822814    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:30:59.825961 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:30:59.825936    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac57b9c_b796_47fb_8538_d3798418a6b7.slice/crio-34fc065fed3f93b7c5468e0841050dc9dcf94a962d63dfe21fac4ba5e3221e16 WatchSource:0}: Error finding container 34fc065fed3f93b7c5468e0841050dc9dcf94a962d63dfe21fac4ba5e3221e16: Status 404 returned error can't find the container with id 34fc065fed3f93b7c5468e0841050dc9dcf94a962d63dfe21fac4ba5e3221e16
Nov 03 20:31:00.590134 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:00.590101    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84d59654-mhsd5" event={"ID":"eac57b9c-b796-47fb-8538-d3798418a6b7","Type":"ContainerStarted","Data":"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"}
Nov 03 20:31:00.590313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:00.590147    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84d59654-mhsd5" event={"ID":"eac57b9c-b796-47fb-8538-d3798418a6b7","Type":"ContainerStarted","Data":"34fc065fed3f93b7c5468e0841050dc9dcf94a962d63dfe21fac4ba5e3221e16"}
Nov 03 20:31:00.701329 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:00.701275    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f84d59654-mhsd5" podStartSLOduration=1.701260405 podStartE2EDuration="1.701260405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:00.700531301 +0000 UTC m=+85.627101537" watchObservedRunningTime="2026-11-03 20:31:00.701260405 +0000 UTC m=+85.627830641"
Nov 03 20:31:09.461384 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:09.461347    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:31:09.461777 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:09.461411    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:31:09.466142 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:09.466116    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:31:09.621371 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:09.621348    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:31:25.684567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.684535    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76749cb79-md28w"]
Nov 03 20:31:25.687695 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.687678    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725500    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093d903-174d-4539-98bf-55ad96afac22-tmp\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725536    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zr4\" (UniqueName: \"kubernetes.io/projected/0093d903-174d-4539-98bf-55ad96afac22-kube-api-access-d9zr4\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725618    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725710    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-oauth-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725757    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-trusted-ca-bundle\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725781    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-service-ca\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725893 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725870    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-console-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.725931 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.725899    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-oauth-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.784995 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.784969    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76749cb79-md28w"]
Nov 03 20:31:25.826305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826274    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093d903-174d-4539-98bf-55ad96afac22-tmp\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826305    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zr4\" (UniqueName: \"kubernetes.io/projected/0093d903-174d-4539-98bf-55ad96afac22-kube-api-access-d9zr4\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826343    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826368    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-oauth-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826388    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-trusted-ca-bundle\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826404    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-service-ca\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826423    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-console-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826450    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-oauth-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.826894 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.826869    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0093d903-174d-4539-98bf-55ad96afac22-tmp\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.827774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.827746    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-console-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.827890 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.827773    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-trusted-ca-bundle\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.827890 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.827804    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-oauth-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.828071 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.828050    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0093d903-174d-4539-98bf-55ad96afac22-service-ca\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.832740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.832716    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-oauth-config\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.833102 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.832975    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0093d903-174d-4539-98bf-55ad96afac22-console-serving-cert\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.883087 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.883056    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zr4\" (UniqueName: \"kubernetes.io/projected/0093d903-174d-4539-98bf-55ad96afac22-kube-api-access-d9zr4\") pod \"console-76749cb79-md28w\" (UID: \"0093d903-174d-4539-98bf-55ad96afac22\") " pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:25.999204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:25.999123    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:26.361788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:26.361762    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76749cb79-md28w"]
Nov 03 20:31:26.363380 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:31:26.363348    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0093d903_174d_4539_98bf_55ad96afac22.slice/crio-a9c55c7f1e183b4970f73062adb2c9160adb762712ec0f16c0296a4d7754fd5d WatchSource:0}: Error finding container a9c55c7f1e183b4970f73062adb2c9160adb762712ec0f16c0296a4d7754fd5d: Status 404 returned error can't find the container with id a9c55c7f1e183b4970f73062adb2c9160adb762712ec0f16c0296a4d7754fd5d
Nov 03 20:31:26.674693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:26.674600    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76749cb79-md28w" event={"ID":"0093d903-174d-4539-98bf-55ad96afac22","Type":"ContainerStarted","Data":"d5c8644c25cf4bd05ed1028f9bfc49542635313f809c76c8c86313a5f8ed3a72"}
Nov 03 20:31:26.674693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:26.674655    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76749cb79-md28w" event={"ID":"0093d903-174d-4539-98bf-55ad96afac22","Type":"ContainerStarted","Data":"a9c55c7f1e183b4970f73062adb2c9160adb762712ec0f16c0296a4d7754fd5d"}
Nov 03 20:31:26.785222 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:26.785172    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76749cb79-md28w" podStartSLOduration=1.7851587169999998 podStartE2EDuration="1.785158717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:26.784727789 +0000 UTC m=+111.711298051" watchObservedRunningTime="2026-11-03 20:31:26.785158717 +0000 UTC m=+111.711728954"
Nov 03 20:31:27.364234 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:31:27.364200    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:31:36.000214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:36.000184    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:36.000636 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:36.000269    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:36.005191 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:36.005169    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:36.715027 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:36.714995    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76749cb79-md28w"
Nov 03 20:31:36.873758 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:31:36.873725    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:31:57.364667 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:31:57.364589    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:32:01.892816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:01.892764    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f84d59654-mhsd5" podUID="eac57b9c-b796-47fb-8538-d3798418a6b7" containerName="console" containerID="cri-o://61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18" gracePeriod=15
Nov 03 20:32:02.137892 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.137866    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:32:02.175238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175192    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175240    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\" (UniqueName: \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175261    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175397 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175378    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175433 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175417    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175482 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175468    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175531 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175502    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175586 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175532    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\") pod \"eac57b9c-b796-47fb-8538-d3798418a6b7\" (UID: \"eac57b9c-b796-47fb-8538-d3798418a6b7\") "
Nov 03 20:32:02.175641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175580    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:02.175697 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175677    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:02.175835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175812    4274 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-oauth-serving-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.175835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175819    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:02.175963 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175839    4274 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-console-config\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.175963 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175880    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:02.175963 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.175911    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:02.177337 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.177314    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "kube-api-access-pkxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:02.177447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.177426    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:02.177550 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.177536    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config" pod "eac57b9c-b796-47fb-8538-d3798418a6b7" (UID: "eac57b9c-b796-47fb-8538-d3798418a6b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:02.276403 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276373    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxhq\" (UniqueName: \"kubernetes.io/projected/eac57b9c-b796-47fb-8538-d3798418a6b7-kube-api-access-pkxhq\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.276403 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276399    4274 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.276403 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276408    4274 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eac57b9c-b796-47fb-8538-d3798418a6b7-trusted-ca-bundle\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.276585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276417    4274 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-serving-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.276585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276427    4274 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac57b9c-b796-47fb-8538-d3798418a6b7-console-oauth-config\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.276585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.276436    4274 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac57b9c-b796-47fb-8538-d3798418a6b7-tmp\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:02.800442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.800405    4274 generic.go:356] "Generic (PLEG): container finished" podID="eac57b9c-b796-47fb-8538-d3798418a6b7" containerID="61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18" exitCode=0
Nov 03 20:32:02.800609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.800449    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84d59654-mhsd5" event={"ID":"eac57b9c-b796-47fb-8538-d3798418a6b7","Type":"ContainerDied","Data":"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"}
Nov 03 20:32:02.800609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.800472    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84d59654-mhsd5"
Nov 03 20:32:02.800609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.800525    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84d59654-mhsd5" event={"ID":"eac57b9c-b796-47fb-8538-d3798418a6b7","Type":"ContainerDied","Data":"34fc065fed3f93b7c5468e0841050dc9dcf94a962d63dfe21fac4ba5e3221e16"}
Nov 03 20:32:02.800609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.800553    4274 scope.go:122] "RemoveContainer" containerID="61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"
Nov 03 20:32:02.808950 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.808888    4274 scope.go:122] "RemoveContainer" containerID="61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"
Nov 03 20:32:02.809159 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:32:02.809137    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18\": container with ID starting with 61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18 not found: ID does not exist" containerID="61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"
Nov 03 20:32:02.809267 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.809169    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18"} err="failed to get container status \"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18\": rpc error: code = NotFound desc = could not find container \"61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18\": container with ID starting with 61c92e81ef18214298863229e11775ae68f2f38a5d57214a07a5b03682ca2b18 not found: ID does not exist"
Nov 03 20:32:02.952899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.952867    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:32:02.999029 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:02.999004    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f84d59654-mhsd5"]
Nov 03 20:32:03.210852 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:32:03.210790    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac57b9c-b796-47fb-8538-d3798418a6b7" path="/var/lib/kubelet/pods/eac57b9c-b796-47fb-8538-d3798418a6b7/volumes"
Nov 03 20:32:27.363671 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:32:27.363634    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:32:57.363946 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:32:57.363908    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:33:27.364371 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:33:27.364336    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:33:57.364063 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:33:57.364016    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:34:27.363565 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:34:27.363528    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:34:42.861262 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.861225    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"]
Nov 03 20:34:42.861774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.861584    4274 cpu_manager.go:414] "containerMap: removing container" podUID="eac57b9c-b796-47fb-8538-d3798418a6b7" containerName="console"
Nov 03 20:34:42.861774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.861601    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac57b9c-b796-47fb-8538-d3798418a6b7" containerName="console"
Nov 03 20:34:42.861774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.861663    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac57b9c-b796-47fb-8538-d3798418a6b7" containerName="console"
Nov 03 20:34:42.864574 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.864556    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:42.916684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.916660    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:34:42.960012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.959979    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:42.960151 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.960057    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:42.960194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.960160    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfhl\" (UniqueName: \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:42.962938 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.962921    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:34:42.963067 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.962976    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-c55wx\""
Nov 03 20:34:42.964155 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:42.964138    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"]
Nov 03 20:34:43.060611 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.060590    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfhl\" (UniqueName: \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.060719 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.060625    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.060719 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.060662    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.061056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.061019    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.061100 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.061060    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.120064 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.119994    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfhl\" (UniqueName: \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.180249 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.180229    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:34:43.495491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.495415    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:34:43.498855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.498776    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.545671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.545632    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"]
Nov 03 20:34:43.552432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552368    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:34:43.552432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552403    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:34:43.552432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552409    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:34:43.552432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552410    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:34:43.552807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552580    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:34:43.552807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552588    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dddj5\""
Nov 03 20:34:43.552807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.552598    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564409    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564455    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564531    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564575    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zkp\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564628    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564677    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.564880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.564709    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.580940 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:34:43.580911    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b3f64_59bd_467f_bbb8_6f81d2d3354d.slice/crio-a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0 WatchSource:0}: Error finding container a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0: Status 404 returned error can't find the container with id a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0
Nov 03 20:34:43.583118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.583101    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:34:43.597387 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.597363    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:34:43.665590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665559    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665595    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665616    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665638    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665667    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665710    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zkp\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.665762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.665751    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.666349 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.666320    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.668082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.668057    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.668196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.668090    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.668333 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.668313    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.668395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.668378    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.722727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.722698    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zkp\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.722830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.722772    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-52shs\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:43.809937 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:43.809906    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:44.223429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:44.223328    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:34:44.225883 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:34:44.225845    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c38946_f202_4977_a677_1707f341f58a.slice/crio-f93e756190c16bc5f35585ed2cb023f624c73e36d871baba4bcb5a6c531416ae WatchSource:0}: Error finding container f93e756190c16bc5f35585ed2cb023f624c73e36d871baba4bcb5a6c531416ae: Status 404 returned error can't find the container with id f93e756190c16bc5f35585ed2cb023f624c73e36d871baba4bcb5a6c531416ae
Nov 03 20:34:44.310855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:44.310819    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" event={"ID":"a0c38946-f202-4977-a677-1707f341f58a","Type":"ContainerStarted","Data":"f93e756190c16bc5f35585ed2cb023f624c73e36d871baba4bcb5a6c531416ae"}
Nov 03 20:34:44.311917 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:44.311894    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerStarted","Data":"a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0"}
Nov 03 20:34:50.335509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:50.335423    4274 generic.go:356] "Generic (PLEG): container finished" podID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerID="1b99f35863d850e84958eb2cf7e890713cfe38c14403f68284f1279e5a912ec7" exitCode=0
Nov 03 20:34:50.335509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:50.335499    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerDied","Data":"1b99f35863d850e84958eb2cf7e890713cfe38c14403f68284f1279e5a912ec7"}
Nov 03 20:34:50.459756 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:50.459728    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:34:50.459835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:50.459793    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:34:51.339627 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:51.339588    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" event={"ID":"a0c38946-f202-4977-a677-1707f341f58a","Type":"ContainerStarted","Data":"a7fe163db21143c40e16f3553eb5b045df35504858dc2900a5d98e290d7b54b2"}
Nov 03 20:34:51.339991 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:51.339798    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:51.341312 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:51.341287    4274 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-52shs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:34:51.341428 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:51.341332    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:34:52.344021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:52.343987    4274 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-52shs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:34:52.344388 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:52.344067    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:34:54.352670 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:54.352633    4274 generic.go:356] "Generic (PLEG): container finished" podID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerID="061b2c55dd5d8fc1678a2e520007ad20c3fb4f44ba7ab455d143590966adab94" exitCode=0
Nov 03 20:34:54.353094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:54.352723    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerDied","Data":"061b2c55dd5d8fc1678a2e520007ad20c3fb4f44ba7ab455d143590966adab94"}
Nov 03 20:34:54.465804 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:54.465749    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" podStartSLOduration=5.235375547 podStartE2EDuration="11.465729667s" totalImagesPullingTime="6.23035412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:51.531312764 +0000 UTC m=+316.457883007" watchObservedRunningTime="2026-11-03 20:34:54.465729667 +0000 UTC m=+319.392299906"
Nov 03 20:34:55.344084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:34:55.344058    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:34:57.367669 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:34:57.367635    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:35:01.378580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:01.378542    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerStarted","Data":"aa2f31dda9eab444da5a1ca2bf17a28390632e21525fdbf289a4f953fd59a65e"}
Nov 03 20:35:01.489803 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:01.489751    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" podStartSLOduration=3.066788332 podStartE2EDuration="19.489738796s" totalImagesPullingTime="16.422950464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:01.488719895 +0000 UTC m=+326.415290132" watchObservedRunningTime="2026-11-03 20:35:01.489738796 +0000 UTC m=+326.416309032"
Nov 03 20:35:02.384007 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:02.383970    4274 generic.go:356] "Generic (PLEG): container finished" podID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerID="aa2f31dda9eab444da5a1ca2bf17a28390632e21525fdbf289a4f953fd59a65e" exitCode=0
Nov 03 20:35:02.384399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:02.384027    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerDied","Data":"aa2f31dda9eab444da5a1ca2bf17a28390632e21525fdbf289a4f953fd59a65e"}
Nov 03 20:35:03.513936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.513911    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:35:03.530736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.530712    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\" (UniqueName: \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\") pod \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") "
Nov 03 20:35:03.530839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.530769    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\") pod \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") "
Nov 03 20:35:03.530910 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.530889    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\") pod \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\" (UID: \"f06b3f64-59bd-467f-bbb8-6f81d2d3354d\") "
Nov 03 20:35:03.531463 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.531436    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle" pod "f06b3f64-59bd-467f-bbb8-6f81d2d3354d" (UID: "f06b3f64-59bd-467f-bbb8-6f81d2d3354d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:03.532998 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.532976    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl" pod "f06b3f64-59bd-467f-bbb8-6f81d2d3354d" (UID: "f06b3f64-59bd-467f-bbb8-6f81d2d3354d"). InnerVolumeSpecName "kube-api-access-xvfhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:03.535255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.535236    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util" pod "f06b3f64-59bd-467f-bbb8-6f81d2d3354d" (UID: "f06b3f64-59bd-467f-bbb8-6f81d2d3354d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:03.632364 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.632330    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfhl\" (UniqueName: \"kubernetes.io/projected/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-kube-api-access-xvfhl\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:03.632364 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.632363    4274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-util\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:03.632364 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:03.632373    4274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f06b3f64-59bd-467f-bbb8-6f81d2d3354d-bundle\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:04.392265 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:04.392230    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff" event={"ID":"f06b3f64-59bd-467f-bbb8-6f81d2d3354d","Type":"ContainerDied","Data":"a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0"}
Nov 03 20:35:04.392447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:04.392280    4274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01ccad75c5adb1c5a114cba766152d460a11cbf34c77859eae0254f8f730bd0"
Nov 03 20:35:04.392447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:04.392237    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb9sff"
Nov 03 20:35:27.364089 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:35:27.364051    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:35:35.716051 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716004    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"]
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716274    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="util"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716286    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="util"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716306    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="extract"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716311    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="extract"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716321    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="pull"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716327    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="pull"
Nov 03 20:35:35.716733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.716374    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06b3f64-59bd-467f-bbb8-6f81d2d3354d" containerName="extract"
Nov 03 20:35:35.719500 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.719481    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757566 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757543    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757574    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757591    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757656    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757801 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757745    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbnm\" (UniqueName: \"kubernetes.io/projected/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-kube-api-access-vdbnm\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.757843 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.757796    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.820350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.820313    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:35:35.820655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.820631    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:35:35.820655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.820647    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:35:35.820787 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.820650    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:35:35.820787 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.820733    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:35:35.837794 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.837770    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fplxc\""
Nov 03 20:35:35.859129 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859105    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbnm\" (UniqueName: \"kubernetes.io/projected/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-kube-api-access-vdbnm\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859136    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859162    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859196    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859214    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859244    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.859666 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.859646    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.861657 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.861633    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.861906 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.861890    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.862094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.862074    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.862151 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.862109    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.918430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.918407    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbnm\" (UniqueName: \"kubernetes.io/projected/8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104-kube-api-access-vdbnm\") pod \"opendatahub-operator-controller-manager-6f448d84c4-dk82h\" (UID: \"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:35.970389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:35.970318    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"]
Nov 03 20:35:36.033492 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:36.033464    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:36.531645 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:36.531606    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"]
Nov 03 20:35:36.535921 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:35:36.535893    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3aee7c_9e8c_4b1e_a92d_6b3d5321f104.slice/crio-63dc48f526b05a3c9b2783c7156088b444e6393f6c16077fe6d81c41e570a695 WatchSource:0}: Error finding container 63dc48f526b05a3c9b2783c7156088b444e6393f6c16077fe6d81c41e570a695: Status 404 returned error can't find the container with id 63dc48f526b05a3c9b2783c7156088b444e6393f6c16077fe6d81c41e570a695
Nov 03 20:35:37.505005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:37.504962    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h" event={"ID":"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104","Type":"ContainerStarted","Data":"63dc48f526b05a3c9b2783c7156088b444e6393f6c16077fe6d81c41e570a695"}
Nov 03 20:35:40.518329 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:40.518296    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h" event={"ID":"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104","Type":"ContainerStarted","Data":"455b3de7b47f864aad81e0a547cb6c9cddd1738ad400638073bb7f7f77cf3033"}
Nov 03 20:35:41.523288 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:41.523252    4274 generic.go:356] "Generic (PLEG): container finished" podID="8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104" containerID="455b3de7b47f864aad81e0a547cb6c9cddd1738ad400638073bb7f7f77cf3033" exitCode=0
Nov 03 20:35:41.523747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:41.523299    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h" event={"ID":"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104","Type":"ContainerDied","Data":"455b3de7b47f864aad81e0a547cb6c9cddd1738ad400638073bb7f7f77cf3033"}
Nov 03 20:35:42.528585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:42.528556    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h" event={"ID":"8c3aee7c-9e8c-4b1e-a92d-6b3d5321f104","Type":"ContainerStarted","Data":"29a089e00cd5697195106b2cc2485b596920a5223d8ee7c27868347761348ccd"}
Nov 03 20:35:42.528929 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:42.528653    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:42.737280 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:42.737190    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h" podStartSLOduration=2.656239954 podStartE2EDuration="7.737177612s" totalImagesPullingTime="4.588676417s" totalInitContainerRuntime="492.261241ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:42.72356246 +0000 UTC m=+367.650132697" watchObservedRunningTime="2026-11-03 20:35:42.737177612 +0000 UTC m=+367.663747849"
Nov 03 20:35:53.535052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:35:53.535003    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-dk82h"
Nov 03 20:35:57.364307 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:35:57.364268    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:36:01.013356 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.013313    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"]
Nov 03 20:36:01.018179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.018158    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.082504 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.082480    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"]
Nov 03 20:36:01.127407 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.127380    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 20:36:01.127538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.127438    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 20:36:01.127538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.127497    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vprwc\""
Nov 03 20:36:01.153623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.153599    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.153745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.153641    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v7wm\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-kube-api-access-7v7wm\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.254349 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.254320    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.254485 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.254356    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v7wm\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-kube-api-access-7v7wm\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.312887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.312820    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.312887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.312879    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v7wm\" (UniqueName: \"kubernetes.io/projected/315b5e64-6ed4-4a35-9d13-aae2b41ff3dc-kube-api-access-7v7wm\") pod \"cert-manager-cainjector-5c57bbd4fc-dst6p\" (UID: \"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.346361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.346336    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"
Nov 03 20:36:01.708698 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:01.708612    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p"]
Nov 03 20:36:01.712070 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:36:01.712020    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod315b5e64_6ed4_4a35_9d13_aae2b41ff3dc.slice/crio-c9ea713681bd20668439b4120778c8768d957704cba48274a834379e7aa8c0a7 WatchSource:0}: Error finding container c9ea713681bd20668439b4120778c8768d957704cba48274a834379e7aa8c0a7: Status 404 returned error can't find the container with id c9ea713681bd20668439b4120778c8768d957704cba48274a834379e7aa8c0a7
Nov 03 20:36:02.601904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:02.601868    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p" event={"ID":"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc","Type":"ContainerStarted","Data":"c9ea713681bd20668439b4120778c8768d957704cba48274a834379e7aa8c0a7"}
Nov 03 20:36:05.614722 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:05.614684    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p" event={"ID":"315b5e64-6ed4-4a35-9d13-aae2b41ff3dc","Type":"ContainerStarted","Data":"ae3a81cd6c197328fec9d6476afd3e46eacb151d60732dcdccd9ef8d58df78ef"}
Nov 03 20:36:05.724591 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:05.724538    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dst6p" podStartSLOduration=2.3192985520000002 podStartE2EDuration="5.724523574s" totalImagesPullingTime="3.405225022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:05.723176952 +0000 UTC m=+390.649747189" watchObservedRunningTime="2026-11-03 20:36:05.724523574 +0000 UTC m=+390.651093815"
Nov 03 20:36:17.083472 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.083396    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"]
Nov 03 20:36:17.086495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.086479    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.181518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.181488    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699e447f-e74f-4d47-ade7-df34275ab26a-tmp\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.181679 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.181528    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699e447f-e74f-4d47-ade7-df34275ab26a-tls-certs\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.181679 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.181545    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmc2s\" (UniqueName: \"kubernetes.io/projected/699e447f-e74f-4d47-ade7-df34275ab26a-kube-api-access-hmc2s\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.183161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.183136    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"]
Nov 03 20:36:17.227556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.227516    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 20:36:17.238383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.238348    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 20:36:17.238383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.238361    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ntmpg\""
Nov 03 20:36:17.282724 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.282697    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699e447f-e74f-4d47-ade7-df34275ab26a-tmp\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.282862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.282731    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699e447f-e74f-4d47-ade7-df34275ab26a-tls-certs\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.282862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.282769    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmc2s\" (UniqueName: \"kubernetes.io/projected/699e447f-e74f-4d47-ade7-df34275ab26a-kube-api-access-hmc2s\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.284859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.284834    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/699e447f-e74f-4d47-ade7-df34275ab26a-tmp\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.285183 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.285164    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699e447f-e74f-4d47-ade7-df34275ab26a-tls-certs\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.339645 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.339592    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmc2s\" (UniqueName: \"kubernetes.io/projected/699e447f-e74f-4d47-ade7-df34275ab26a-kube-api-access-hmc2s\") pod \"kube-auth-proxy-7ccb65679-p4k6x\" (UID: \"699e447f-e74f-4d47-ade7-df34275ab26a\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.398355 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.398329    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"
Nov 03 20:36:17.761863 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:17.761830    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x"]
Nov 03 20:36:17.764548 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:36:17.764523    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699e447f_e74f_4d47_ade7_df34275ab26a.slice/crio-1c1a4f8b499b19d19c3d2ec75071bb1b550ba08ae90426b87264ed451433970a WatchSource:0}: Error finding container 1c1a4f8b499b19d19c3d2ec75071bb1b550ba08ae90426b87264ed451433970a: Status 404 returned error can't find the container with id 1c1a4f8b499b19d19c3d2ec75071bb1b550ba08ae90426b87264ed451433970a
Nov 03 20:36:18.660496 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:18.660459    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x" event={"ID":"699e447f-e74f-4d47-ade7-df34275ab26a","Type":"ContainerStarted","Data":"1c1a4f8b499b19d19c3d2ec75071bb1b550ba08ae90426b87264ed451433970a"}
Nov 03 20:36:22.678350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:22.678310    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x" event={"ID":"699e447f-e74f-4d47-ade7-df34275ab26a","Type":"ContainerStarted","Data":"60699f15c020bffd88d917d9302d7abcc2fc6a5162337e0091e72afd77c7801f"}
Nov 03 20:36:22.790151 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:22.790095    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ccb65679-p4k6x" podStartSLOduration=1.891672973 podStartE2EDuration="5.790080707s" totalImagesPullingTime="3.898407734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:22.788800626 +0000 UTC m=+407.715370864" watchObservedRunningTime="2026-11-03 20:36:22.790080707 +0000 UTC m=+407.716650943"
Nov 03 20:36:27.364367 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:36:27.364324    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:36:47.922282 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:47.922201    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"]
Nov 03 20:36:47.925368 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:47.925348    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.001688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.001654    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-cert\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.001688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.001691    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bad9a476-500f-4a0c-b668-5ba3bcec2522-manager-config\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.001845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.001707    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/bad9a476-500f-4a0c-b668-5ba3bcec2522-kube-api-access-lfxmj\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.001845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.001729    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-metrics-certs\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.020469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.020439    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 20:36:48.020469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.020455    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 20:36:48.021539 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.021519    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"]
Nov 03 20:36:48.031450 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.031433    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-gb76r\""
Nov 03 20:36:48.031450 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.031441    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 20:36:48.031594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.031441    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:36:48.031594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.031458    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 20:36:48.103167 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.103135    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-cert\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.103309 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.103176    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bad9a476-500f-4a0c-b668-5ba3bcec2522-manager-config\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.103309 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.103202    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/bad9a476-500f-4a0c-b668-5ba3bcec2522-kube-api-access-lfxmj\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.103309 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.103232    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-metrics-certs\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.104010 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.103869    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bad9a476-500f-4a0c-b668-5ba3bcec2522-manager-config\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.111684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.111656    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-cert\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.111812 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.111664    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bad9a476-500f-4a0c-b668-5ba3bcec2522-metrics-certs\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.160322 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.160293    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxmj\" (UniqueName: \"kubernetes.io/projected/bad9a476-500f-4a0c-b668-5ba3bcec2522-kube-api-access-lfxmj\") pod \"jobset-controller-manager-756876c46b-npxc6\" (UID: \"bad9a476-500f-4a0c-b668-5ba3bcec2522\") " pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.237265 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.237189    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:48.599617 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.599589    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"]
Nov 03 20:36:48.602230 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:36:48.602194    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad9a476_500f_4a0c_b668_5ba3bcec2522.slice/crio-ed5bc54fe27366c2e87c9ca71b4597e198d048718723c41b85e785d41bbabcb2 WatchSource:0}: Error finding container ed5bc54fe27366c2e87c9ca71b4597e198d048718723c41b85e785d41bbabcb2: Status 404 returned error can't find the container with id ed5bc54fe27366c2e87c9ca71b4597e198d048718723c41b85e785d41bbabcb2
Nov 03 20:36:48.775686 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:48.775651    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6" event={"ID":"bad9a476-500f-4a0c-b668-5ba3bcec2522","Type":"ContainerStarted","Data":"ed5bc54fe27366c2e87c9ca71b4597e198d048718723c41b85e785d41bbabcb2"}
Nov 03 20:36:56.807064 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:56.807016    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6" event={"ID":"bad9a476-500f-4a0c-b668-5ba3bcec2522","Type":"ContainerStarted","Data":"31303dd513eb08d83f20a27f4d0d98d9fc54e8d90f42c75f8b3f5b5185de3b26"}
Nov 03 20:36:56.807481 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:56.807166    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:36:56.917008 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:36:56.916954    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6" podStartSLOduration=2.527627794 podStartE2EDuration="9.916941244s" totalImagesPullingTime="7.38931345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:56.916360944 +0000 UTC m=+441.842931180" watchObservedRunningTime="2026-11-03 20:36:56.916941244 +0000 UTC m=+441.843511480"
Nov 03 20:36:57.364489 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:36:57.364454    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:37:07.817418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:07.817388    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-756876c46b-npxc6"
Nov 03 20:37:27.364360 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:37:27.364321    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:37:45.726382 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:45.726313    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:37:45.726823 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:45.726501    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery" containerID="cri-o://a7fe163db21143c40e16f3553eb5b045df35504858dc2900a5d98e290d7b54b2" gracePeriod=30
Nov 03 20:37:45.989468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:45.989387    4274 generic.go:356] "Generic (PLEG): container finished" podID="a0c38946-f202-4977-a677-1707f341f58a" containerID="a7fe163db21143c40e16f3553eb5b045df35504858dc2900a5d98e290d7b54b2" exitCode=0
Nov 03 20:37:45.989468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:45.989452    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" event={"ID":"a0c38946-f202-4977-a677-1707f341f58a","Type":"ContainerDied","Data":"a7fe163db21143c40e16f3553eb5b045df35504858dc2900a5d98e290d7b54b2"}
Nov 03 20:37:46.066527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.066499    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:37:46.260878 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.260856    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.260889    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.260919    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.260952    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.260995    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261052 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.261021    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261377 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.261105    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\") pod \"a0c38946-f202-4977-a677-1707f341f58a\" (UID: \"a0c38946-f202-4977-a677-1707f341f58a\") "
Nov 03 20:37:46.261377 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.261343    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:46.263412 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263379    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:46.263538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263440    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "kube-api-access-t5zkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:46.263588 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263558    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:46.263639 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263586    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:46.263674 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263654    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:46.263772 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.263756    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert" pod "a0c38946-f202-4977-a677-1707f341f58a" (UID: "a0c38946-f202-4977-a677-1707f341f58a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:46.362192 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362162    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zkp\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-kube-api-access-t5zkp\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362371 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362338    4274 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-ca-configmap\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362371 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362368    4274 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-csr-dns-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362384    4274 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-cacerts\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362402    4274 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c38946-f202-4977-a677-1707f341f58a-local-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362416    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c38946-f202-4977-a677-1707f341f58a-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.362509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.362431    4274 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0c38946-f202-4977-a677-1707f341f58a-istio-kubeconfig\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.995131 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.995102    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"
Nov 03 20:37:46.995131 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.995124    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs" event={"ID":"a0c38946-f202-4977-a677-1707f341f58a","Type":"ContainerDied","Data":"f93e756190c16bc5f35585ed2cb023f624c73e36d871baba4bcb5a6c531416ae"}
Nov 03 20:37:46.995598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:46.995175    4274 scope.go:122] "RemoveContainer" containerID="a7fe163db21143c40e16f3553eb5b045df35504858dc2900a5d98e290d7b54b2"
Nov 03 20:37:47.225659 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:47.225632    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:37:47.271272 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:47.271248    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-52shs"]
Nov 03 20:37:49.211333 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:49.211296    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c38946-f202-4977-a677-1707f341f58a" path="/var/lib/kubelet/pods/a0c38946-f202-4977-a677-1707f341f58a/volumes"
Nov 03 20:37:55.164432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.164400    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:37:55.165005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.164774    4274 cpu_manager.go:414] "containerMap: removing container" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery"
Nov 03 20:37:55.165005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.164787    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery"
Nov 03 20:37:55.165005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.164879    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c38946-f202-4977-a677-1707f341f58a" containerName="discovery"
Nov 03 20:37:55.169015 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.168997    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:37:55.221673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.221649    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4lx\" (UniqueName: \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\") pod \"authorino-6f6944bcdd-nxkrl\" (UID: \"1e95aed7-59d5-41f7-947a-a5fe16eca011\") " pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:37:55.228532 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.228509    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4v8fx\""
Nov 03 20:37:55.269280 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.269252    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:37:55.269435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.269327    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:37:55.270385 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.270366    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:37:55.322803 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.322774    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4lx\" (UniqueName: \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\") pod \"authorino-6f6944bcdd-nxkrl\" (UID: \"1e95aed7-59d5-41f7-947a-a5fe16eca011\") " pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:37:55.403253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.403227    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 20:37:55.406461 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.406439    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.409556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.409534    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4lx\" (UniqueName: \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\") pod \"authorino-6f6944bcdd-nxkrl\" (UID: \"1e95aed7-59d5-41f7-947a-a5fe16eca011\") " pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:37:55.423442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.423421    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.423530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.423448    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.423530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.423489    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmdb\" (UniqueName: \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.423621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.423599    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.460504 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.460481    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:37:55.460609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.460588    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:37:55.460609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.460589    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:37:55.460707 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.460603    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\""
Nov 03 20:37:55.460707 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.460647    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:37:55.466619 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.466590    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 20:37:55.479122 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.479104    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:37:55.528524 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.528497    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmdb\" (UniqueName: \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.528620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.528547    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.528620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.528584    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.528620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.528601    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.531224 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.531196    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.532382 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.532347    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.533327 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.533304    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.544437 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.544411    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:37:55.580179 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.580149    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b96f79648-jcn8w"]
Nov 03 20:37:55.584528 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.584509    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:55.595061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.595027    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmdb\" (UniqueName: \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\") pod \"maas-api-844bc99d-bp7x7\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.645709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.645684    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b96f79648-jcn8w"]
Nov 03 20:37:55.725488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.725430    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:37:55.730496 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.730474    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjh78\" (UniqueName: \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\") pod \"authorino-6b96f79648-jcn8w\" (UID: \"c6344f5f-75dc-4a07-bd17-5408e1c37526\") " pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:55.831545 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.831505    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjh78\" (UniqueName: \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\") pod \"authorino-6b96f79648-jcn8w\" (UID: \"c6344f5f-75dc-4a07-bd17-5408e1c37526\") " pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:55.839784 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.839760    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:37:55.842174 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:37:55.842150    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e95aed7_59d5_41f7_947a_a5fe16eca011.slice/crio-5cde5cbc630f4c80436d2c58c33335c6452d59e7d1c059dd2617764a6c9635d0 WatchSource:0}: Error finding container 5cde5cbc630f4c80436d2c58c33335c6452d59e7d1c059dd2617764a6c9635d0: Status 404 returned error can't find the container with id 5cde5cbc630f4c80436d2c58c33335c6452d59e7d1c059dd2617764a6c9635d0
Nov 03 20:37:55.882616 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.882583    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b96f79648-jcn8w"]
Nov 03 20:37:55.882798 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:37:55.882778    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sjh78], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6b96f79648-jcn8w" podUID="c6344f5f-75dc-4a07-bd17-5408e1c37526"
Nov 03 20:37:55.897345 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.897317    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjh78\" (UniqueName: \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\") pod \"authorino-6b96f79648-jcn8w\" (UID: \"c6344f5f-75dc-4a07-bd17-5408e1c37526\") " pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:55.913145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.913120    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:37:55.917562 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.917545    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:55.932807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.932785    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw7q\" (UniqueName: \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:55.932872 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.932850    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:55.966899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.966877    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 20:37:55.972062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:55.972022    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:37:56.033438 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.033412    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:56.033621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.033467    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw7q\" (UniqueName: \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:56.036067 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.036021    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:56.036322 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.036301    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:56.036425 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.036320    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" event={"ID":"1e95aed7-59d5-41f7-947a-a5fe16eca011","Type":"ContainerStarted","Data":"5cde5cbc630f4c80436d2c58c33335c6452d59e7d1c059dd2617764a6c9635d0"}
Nov 03 20:37:56.042442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.042426    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:56.091332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.091313    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw7q\" (UniqueName: \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\") pod \"authorino-687fd5c7fd-gxmns\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") " pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:56.093959 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.093943    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 20:37:56.095853 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:37:56.095830    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f12b1_2087_4bb2_92d5_126e0d154b0f.slice/crio-6897fd77084f26f1fa53958d818d3a2e43e2ace9a397e0ffb16935ee4b710ffc WatchSource:0}: Error finding container 6897fd77084f26f1fa53958d818d3a2e43e2ace9a397e0ffb16935ee4b710ffc: Status 404 returned error can't find the container with id 6897fd77084f26f1fa53958d818d3a2e43e2ace9a397e0ffb16935ee4b710ffc
Nov 03 20:37:56.134068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.134031    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\" (UniqueName: \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\") pod \"c6344f5f-75dc-4a07-bd17-5408e1c37526\" (UID: \"c6344f5f-75dc-4a07-bd17-5408e1c37526\") "
Nov 03 20:37:56.135856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.135837    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78" pod "c6344f5f-75dc-4a07-bd17-5408e1c37526" (UID: "c6344f5f-75dc-4a07-bd17-5408e1c37526"). InnerVolumeSpecName "kube-api-access-sjh78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:56.228104 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.228081    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:37:56.234552 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.234530    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjh78\" (UniqueName: \"kubernetes.io/projected/c6344f5f-75dc-4a07-bd17-5408e1c37526-kube-api-access-sjh78\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:56.589439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:56.589416    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:37:56.591343 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:37:56.591311    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ff5caf_b24d_4bd4_836b_f370e05b9c95.slice/crio-8ef3e540650297e8691ecbd46e8bfd0950d3d5a3b4b0518bbb139315c43812a5 WatchSource:0}: Error finding container 8ef3e540650297e8691ecbd46e8bfd0950d3d5a3b4b0518bbb139315c43812a5: Status 404 returned error can't find the container with id 8ef3e540650297e8691ecbd46e8bfd0950d3d5a3b4b0518bbb139315c43812a5
Nov 03 20:37:57.042017 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:57.041975    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" event={"ID":"94ff5caf-b24d-4bd4-836b-f370e05b9c95","Type":"ContainerStarted","Data":"8ef3e540650297e8691ecbd46e8bfd0950d3d5a3b4b0518bbb139315c43812a5"}
Nov 03 20:37:57.043330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:57.043300    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" event={"ID":"3b2f12b1-2087-4bb2-92d5-126e0d154b0f","Type":"ContainerStarted","Data":"6897fd77084f26f1fa53958d818d3a2e43e2ace9a397e0ffb16935ee4b710ffc"}
Nov 03 20:37:57.043544 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:57.043344    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96f79648-jcn8w"
Nov 03 20:37:57.267291 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:57.267222    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b96f79648-jcn8w"]
Nov 03 20:37:57.270262 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:57.270237    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b96f79648-jcn8w"]
Nov 03 20:37:57.371553 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:37:57.371481    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:37:59.213059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:37:59.213013    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6344f5f-75dc-4a07-bd17-5408e1c37526" path="/var/lib/kubelet/pods/c6344f5f-75dc-4a07-bd17-5408e1c37526/volumes"
Nov 03 20:38:00.063396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.063354    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" event={"ID":"1e95aed7-59d5-41f7-947a-a5fe16eca011","Type":"ContainerStarted","Data":"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"}
Nov 03 20:38:00.063577 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.063417    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" podUID="1e95aed7-59d5-41f7-947a-a5fe16eca011" containerName="authorino" containerID="cri-o://d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793" gracePeriod=30
Nov 03 20:38:00.065085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.065062    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" event={"ID":"3b2f12b1-2087-4bb2-92d5-126e0d154b0f","Type":"ContainerStarted","Data":"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"}
Nov 03 20:38:00.065194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.065152    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:38:00.296839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.296794    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" podStartSLOduration=2.22529875 podStartE2EDuration="5.296780145s" totalImagesPullingTime="3.071481395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:00.295961866 +0000 UTC m=+505.222532102" watchObservedRunningTime="2026-11-03 20:38:00.296780145 +0000 UTC m=+505.223350381"
Nov 03 20:38:00.297481 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.297438    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" podStartSLOduration=1.570749262 podStartE2EDuration="5.297425907s" totalImagesPullingTime="3.726676645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:00.181802777 +0000 UTC m=+505.108373027" watchObservedRunningTime="2026-11-03 20:38:00.297425907 +0000 UTC m=+505.223996146"
Nov 03 20:38:00.309586 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.309566    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:38:00.365839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.365785    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\" (UniqueName: \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\") pod \"1e95aed7-59d5-41f7-947a-a5fe16eca011\" (UID: \"1e95aed7-59d5-41f7-947a-a5fe16eca011\") "
Nov 03 20:38:00.367824 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.367799    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx" pod "1e95aed7-59d5-41f7-947a-a5fe16eca011" (UID: "1e95aed7-59d5-41f7-947a-a5fe16eca011"). InnerVolumeSpecName "kube-api-access-5f4lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:00.466649 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:00.466618    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4lx\" (UniqueName: \"kubernetes.io/projected/1e95aed7-59d5-41f7-947a-a5fe16eca011-kube-api-access-5f4lx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:01.070398 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.070363    4274 generic.go:356] "Generic (PLEG): container finished" podID="1e95aed7-59d5-41f7-947a-a5fe16eca011" containerID="d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793" exitCode=0
Nov 03 20:38:01.070584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.070412    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl"
Nov 03 20:38:01.070584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.070412    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" event={"ID":"1e95aed7-59d5-41f7-947a-a5fe16eca011","Type":"ContainerDied","Data":"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"}
Nov 03 20:38:01.070584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.070521    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-nxkrl" event={"ID":"1e95aed7-59d5-41f7-947a-a5fe16eca011","Type":"ContainerDied","Data":"5cde5cbc630f4c80436d2c58c33335c6452d59e7d1c059dd2617764a6c9635d0"}
Nov 03 20:38:01.070584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.070542    4274 scope.go:122] "RemoveContainer" containerID="d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"
Nov 03 20:38:01.079102 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.079082    4274 scope.go:122] "RemoveContainer" containerID="d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"
Nov 03 20:38:01.079371 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:38:01.079344    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793\": container with ID starting with d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793 not found: ID does not exist" containerID="d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"
Nov 03 20:38:01.079457 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.079376    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793"} err="failed to get container status \"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793\": rpc error: code = NotFound desc = could not find container \"d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793\": container with ID starting with d99430e4c4fc19197ccb74b4bb43a39b4163b7f36debfa023d666b3952457793 not found: ID does not exist"
Nov 03 20:38:01.223295 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.223263    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:38:01.226529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:01.226492    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nxkrl"]
Nov 03 20:38:03.080383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:03.080350    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" event={"ID":"94ff5caf-b24d-4bd4-836b-f370e05b9c95","Type":"ContainerStarted","Data":"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"}
Nov 03 20:38:03.187848 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:03.187807    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" podStartSLOduration=2.188472946 podStartE2EDuration="8.187794102s" totalImagesPullingTime="5.999321156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:03.186924677 +0000 UTC m=+508.113494913" watchObservedRunningTime="2026-11-03 20:38:03.187794102 +0000 UTC m=+508.114364338"
Nov 03 20:38:03.214336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:03.214303    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e95aed7-59d5-41f7-947a-a5fe16eca011" path="/var/lib/kubelet/pods/1e95aed7-59d5-41f7-947a-a5fe16eca011/volumes"
Nov 03 20:38:06.076227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:38:06.076200    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 20:38:27.364793 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:38:27.364753    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:38:57.364876 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:38:57.364775    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:39:27.363671 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:39:27.363637    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:39:43.489302 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.489271    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"]
Nov 03 20:39:43.489918 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.489631    4274 cpu_manager.go:414] "containerMap: removing container" podUID="1e95aed7-59d5-41f7-947a-a5fe16eca011" containerName="authorino"
Nov 03 20:39:43.489918 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.489646    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e95aed7-59d5-41f7-947a-a5fe16eca011" containerName="authorino"
Nov 03 20:39:43.489918 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.489722    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e95aed7-59d5-41f7-947a-a5fe16eca011" containerName="authorino"
Nov 03 20:39:43.492514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.492498    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.541885 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.541857    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.542029 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.541910    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klcr\" (UniqueName: \"kubernetes.io/projected/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-kube-api-access-6klcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.542029 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.542000    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.542151 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.542063    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.542191 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.542151    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.542239 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.542203    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.587683 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.587654    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zkx7m\""
Nov 03 20:39:43.587683 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.587656    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:39:43.587859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.587724    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:39:43.587859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.587745    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 20:39:43.589500 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.589474    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"]
Nov 03 20:39:43.643236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643209    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643362 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643273    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643362 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643325    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klcr\" (UniqueName: \"kubernetes.io/projected/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-kube-api-access-6klcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643362 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643357    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643391    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643432    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643636 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643608    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643749 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643731    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.643806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.643764    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.645458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.645439    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.645618 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.645601    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.705262 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.705236    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klcr\" (UniqueName: \"kubernetes.io/projected/64cb18ce-10a1-465c-9eec-ebf0cd65ffc2-kube-api-access-6klcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw\" (UID: \"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:43.804916 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:43.804889    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:44.171799 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:44.171731    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"]
Nov 03 20:39:44.174512 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:39:44.174484    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb18ce_10a1_465c_9eec_ebf0cd65ffc2.slice/crio-c2d9aaf4fa3d0b93e7dc0bddc616cee2c59ea3777db8de24b7ed0153187caf9f WatchSource:0}: Error finding container c2d9aaf4fa3d0b93e7dc0bddc616cee2c59ea3777db8de24b7ed0153187caf9f: Status 404 returned error can't find the container with id c2d9aaf4fa3d0b93e7dc0bddc616cee2c59ea3777db8de24b7ed0153187caf9f
Nov 03 20:39:44.175788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:44.175767    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:39:44.456058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:44.455968    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw" event={"ID":"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2","Type":"ContainerStarted","Data":"c2d9aaf4fa3d0b93e7dc0bddc616cee2c59ea3777db8de24b7ed0153187caf9f"}
Nov 03 20:39:46.465142 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:46.465107    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw" event={"ID":"64cb18ce-10a1-465c-9eec-ebf0cd65ffc2","Type":"ContainerStarted","Data":"af3924708c576bdee94bf91744d4307531042649be36c0f95d4887a81c8e176c"}
Nov 03 20:39:46.575933 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:46.575883    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw" podStartSLOduration=2.039085092 podStartE2EDuration="3.575869083s" totalImagesPullingTime="1.536783991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:46.574920515 +0000 UTC m=+611.501490751" watchObservedRunningTime="2026-11-03 20:39:46.575869083 +0000 UTC m=+611.502439319"
Nov 03 20:39:46.804998 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:46.804961    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:47.509949 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.509908    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"]
Nov 03 20:39:47.524201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.524174    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573070 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573017    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573084    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573186    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573244    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573497 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573470    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6l86\" (UniqueName: \"kubernetes.io/projected/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-kube-api-access-q6l86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.573567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.573551    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.619054 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.619010    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 20:39:47.620338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.620316    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"]
Nov 03 20:39:47.674868 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.674836    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.674868 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.674870    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.674902    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.674940    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.674999    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l86\" (UniqueName: \"kubernetes.io/projected/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-kube-api-access-q6l86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.675030    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675359 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.675318    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675428 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.675403    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.675496 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.675478    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.677141 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.677119    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.677309 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.677293    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.747597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.747559    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l86\" (UniqueName: \"kubernetes.io/projected/8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30-kube-api-access-q6l86\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w\" (UID: \"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:47.838295 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:47.838211    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:48.213462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:48.213373    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"]
Nov 03 20:39:48.215962 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:39:48.215938    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9caaa0_3216_4a6f_90a6_a51cb7b8cd30.slice/crio-db3fe4835eacbd1424ba63801d3577542ac20c5885d52145ff5862cad708c96f WatchSource:0}: Error finding container db3fe4835eacbd1424ba63801d3577542ac20c5885d52145ff5862cad708c96f: Status 404 returned error can't find the container with id db3fe4835eacbd1424ba63801d3577542ac20c5885d52145ff5862cad708c96f
Nov 03 20:39:48.473772 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:48.473681    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w" event={"ID":"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30","Type":"ContainerStarted","Data":"db3fe4835eacbd1424ba63801d3577542ac20c5885d52145ff5862cad708c96f"}
Nov 03 20:39:49.479223 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:49.479186    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w" event={"ID":"8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30","Type":"ContainerStarted","Data":"4b4d058bbdb2de4c95c07ad8e6cbdfea0ff1004cc7c70459cbec4d4f580fa0ed"}
Nov 03 20:39:49.592936 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:49.592726    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w" podStartSLOduration=1.9668369879999998 podStartE2EDuration="2.59271259s" totalImagesPullingTime="625.875602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:49.591454248 +0000 UTC m=+614.518024486" watchObservedRunningTime="2026-11-03 20:39:49.59271259 +0000 UTC m=+614.519282826"
Nov 03 20:39:49.838835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:49.838805    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:50.811439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.811404    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kksgw"]
Nov 03 20:39:50.814701 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.814676    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.901822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.901792    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.901988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.901828    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297kb\" (UniqueName: \"kubernetes.io/projected/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-kube-api-access-297kb\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.901988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.901847    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.901988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.901923    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.902169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.902117    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.902224 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.902170    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:50.909442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.909425    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 20:39:50.910884 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:50.910869    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kksgw"]
Nov 03 20:39:51.003381 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003354    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003398    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003422    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297kb\" (UniqueName: \"kubernetes.io/projected/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-kube-api-access-297kb\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003445    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003502    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003556    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003853 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003831    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003942 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003895    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.003942 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.003913    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.005838 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.005816    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.006016 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.005993    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.060595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.060568    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297kb\" (UniqueName: \"kubernetes.io/projected/be0438e3-7fe9-4c9f-a66e-4a89b436ffeb-kube-api-access-297kb\") pod \"echo-simulator-kserve-64b6f8f96c-kksgw\" (UID: \"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.126741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.126691    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:51.487490 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:51.487412    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kksgw"]
Nov 03 20:39:51.489867 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:39:51.489842    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0438e3_7fe9_4c9f_a66e_4a89b436ffeb.slice/crio-8a0976714fbf2a05009b5ac534a4fce67ee6bd6617699ece50181d789f89c20f WatchSource:0}: Error finding container 8a0976714fbf2a05009b5ac534a4fce67ee6bd6617699ece50181d789f89c20f: Status 404 returned error can't find the container with id 8a0976714fbf2a05009b5ac534a4fce67ee6bd6617699ece50181d789f89c20f
Nov 03 20:39:52.494119 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:52.494088    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw" event={"ID":"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb","Type":"ContainerStarted","Data":"8a0976714fbf2a05009b5ac534a4fce67ee6bd6617699ece50181d789f89c20f"}
Nov 03 20:39:53.806101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:53.806060    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:53.828614 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:53.828358    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:54.511714 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:54.511669    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw" event={"ID":"be0438e3-7fe9-4c9f-a66e-4a89b436ffeb","Type":"ContainerStarted","Data":"d4dc44c16a678f52ca251624affd86ce48a667152a256f41bc7ebb2c354e4a70"}
Nov 03 20:39:54.523811 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:54.523783    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw"
Nov 03 20:39:54.624373 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:54.624313    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw" podStartSLOduration=2.280661742 podStartE2EDuration="4.624298356s" totalImagesPullingTime="2.343636614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:54.62288049 +0000 UTC m=+619.549450742" watchObservedRunningTime="2026-11-03 20:39:54.624298356 +0000 UTC m=+619.550868623"
Nov 03 20:39:55.127621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:55.127584    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:39:57.364467 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:39:57.364436    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:39:57.838534 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:57.838500    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:57.850745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:57.850719    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:58.543491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.543464    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w"
Nov 03 20:39:58.707340 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.707290    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"]
Nov 03 20:39:58.710842 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.710818    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.759727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.759692    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 20:39:58.768059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.768018    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"]
Nov 03 20:39:58.773730 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773704    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgk2\" (UniqueName: \"kubernetes.io/projected/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-kube-api-access-plgk2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.773839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773751    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.773839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773812    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.773839 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773831    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.773941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773857    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.773941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.773887    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875083    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgk2\" (UniqueName: \"kubernetes.io/projected/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-kube-api-access-plgk2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875129    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875151    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875409 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875174    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875409 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875210    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875409 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875365    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875552 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875530    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875602 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875556    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.875646 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.875628    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.877456 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.877431    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.877667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.877649    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:58.931938 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:58.931915    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgk2\" (UniqueName: \"kubernetes.io/projected/2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac-kube-api-access-plgk2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p\" (UID: \"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:59.021699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:59.021677    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:39:59.384966 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:59.384868    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"]
Nov 03 20:39:59.387189 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:39:59.387153    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d4a77bd_c3f6_4bad_9b9b_3f6e4e89f8ac.slice/crio-9d1b52f4cb74c1c392d25ef11cc5c0212fdc73b695d7b9540837f07449df4188 WatchSource:0}: Error finding container 9d1b52f4cb74c1c392d25ef11cc5c0212fdc73b695d7b9540837f07449df4188: Status 404 returned error can't find the container with id 9d1b52f4cb74c1c392d25ef11cc5c0212fdc73b695d7b9540837f07449df4188
Nov 03 20:39:59.533711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:39:59.533677    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" event={"ID":"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac","Type":"ContainerStarted","Data":"9d1b52f4cb74c1c392d25ef11cc5c0212fdc73b695d7b9540837f07449df4188"}
Nov 03 20:40:00.539059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:00.539016    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" event={"ID":"2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac","Type":"ContainerStarted","Data":"336bb5310f2c4e742db43bf26eff386738451b75314c911f19c6d3710bcee1e8"}
Nov 03 20:40:00.649563 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:00.649515    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podStartSLOduration=1.984779139 podStartE2EDuration="2.649501457s" totalImagesPullingTime="664.722318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:00.648903635 +0000 UTC m=+625.575473890" watchObservedRunningTime="2026-11-03 20:40:00.649501457 +0000 UTC m=+625.576071694"
Nov 03 20:40:01.022756 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:01.022733    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:40:01.127546 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:01.127519    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:40:01.139683 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:01.139659    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:40:01.554147 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:01.554121    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kksgw"
Nov 03 20:40:09.022142 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.022109    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:40:09.034287 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.034263    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:40:09.281635 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.281548    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"]
Nov 03 20:40:09.284782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.284767    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.333742 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.333713    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 20:40:09.342732 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.342697    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"]
Nov 03 20:40:09.366478 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366449    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.366623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366488    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.366623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366509    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.366699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366643    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.366699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366668    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlpr\" (UniqueName: \"kubernetes.io/projected/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-kube-api-access-fhlpr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.366764 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.366746    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467290 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467258    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467290 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467295    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467313    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467335    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467351    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlpr\" (UniqueName: \"kubernetes.io/projected/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-kube-api-access-fhlpr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467403    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467746 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467720    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467729    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.467820 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.467792    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.469672 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.469652    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.469961 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.469942    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.524605 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.524585    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlpr\" (UniqueName: \"kubernetes.io/projected/739f1d75-4415-4dd2-bfcc-c3c4f83e653b-kube-api-access-fhlpr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7h9hq\" (UID: \"739f1d75-4415-4dd2-bfcc-c3c4f83e653b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.593380 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.593322    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:40:09.596641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.596620    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:09.962525 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:09.962491    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"]
Nov 03 20:40:09.963923 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:40:09.963899    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739f1d75_4415_4dd2_bfcc_c3c4f83e653b.slice/crio-60e82a3fbf441e24303957f5e7e60e96165d88a10da2b4c9821585b434f9776c WatchSource:0}: Error finding container 60e82a3fbf441e24303957f5e7e60e96165d88a10da2b4c9821585b434f9776c: Status 404 returned error can't find the container with id 60e82a3fbf441e24303957f5e7e60e96165d88a10da2b4c9821585b434f9776c
Nov 03 20:40:10.587462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:10.587424    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq" event={"ID":"739f1d75-4415-4dd2-bfcc-c3c4f83e653b","Type":"ContainerStarted","Data":"60e82a3fbf441e24303957f5e7e60e96165d88a10da2b4c9821585b434f9776c"}
Nov 03 20:40:11.592312 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:11.592282    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq" event={"ID":"739f1d75-4415-4dd2-bfcc-c3c4f83e653b","Type":"ContainerStarted","Data":"954616b8c8d1b007d44cb3983fbb1d6aa4f2fe2e9a3cf6ea68442524a55af2a5"}
Nov 03 20:40:11.596725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:11.596708    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:11.702925 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:11.702872    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq" podStartSLOduration=2.073496962 podStartE2EDuration="2.702859306s" totalImagesPullingTime="629.362344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:11.702643675 +0000 UTC m=+636.629213909" watchObservedRunningTime="2026-11-03 20:40:11.702859306 +0000 UTC m=+636.629429543"
Nov 03 20:40:19.597743 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:19.597710    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:19.609964 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:19.609944    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:19.634421 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:19.634397    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7h9hq"
Nov 03 20:40:25.051957 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.051923    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:40:25.055366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.055345    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.099959 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.099928    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100135 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.099967    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100135 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.099990    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100278 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100131    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100278 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100171    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100278 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100271    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100303    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100331    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.100418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.100356    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkj5k\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.105087 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.105068    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:40:25.151009 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.150986    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\""
Nov 03 20:40:25.155367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.155341    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:40:25.200924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.200895    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.200925    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.200947    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201138    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkj5k\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201254 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201228    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201266    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201282    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201297    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201476 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201373    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201476 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201423    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201556    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201819 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201791    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.201819 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201801    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.202327 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.201930    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.203406 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.203391    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.203816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.203794    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.257978 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.257949    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.258112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.258095    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkj5k\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.367844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.367765    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:25.778931 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:25.778909    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:40:25.781409 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:40:25.781382    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc840bef8_ed6c_473b_99d3_ff601513d093.slice/crio-0f525f6e3763064e0f2047951393ef4ebb0294553c81a982d0dada7f2068fceb WatchSource:0}: Error finding container 0f525f6e3763064e0f2047951393ef4ebb0294553c81a982d0dada7f2068fceb: Status 404 returned error can't find the container with id 0f525f6e3763064e0f2047951393ef4ebb0294553c81a982d0dada7f2068fceb
Nov 03 20:40:26.654185 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:26.654149    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" event={"ID":"c840bef8-ed6c-473b-99d3-ff601513d093","Type":"ContainerStarted","Data":"0f525f6e3763064e0f2047951393ef4ebb0294553c81a982d0dada7f2068fceb"}
Nov 03 20:40:27.364535 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:40:27.364497    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:40:29.758719 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.758686    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-784fb86d65-6kg8v"]
Nov 03 20:40:29.767712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.767686    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:29.817160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.817133    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-784fb86d65-6kg8v"]
Nov 03 20:40:29.843207 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.843179    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjv2f\" (UniqueName: \"kubernetes.io/projected/1a2f424c-7201-4beb-acc4-4481306fb5a3-kube-api-access-bjv2f\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:29.843366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.843219    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a2f424c-7201-4beb-acc4-4481306fb5a3-tls-cert\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:29.937311 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.937282    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:29.937443 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.937361    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:29.937443 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.937406    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:29.944165 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.944139    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjv2f\" (UniqueName: \"kubernetes.io/projected/1a2f424c-7201-4beb-acc4-4481306fb5a3-kube-api-access-bjv2f\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:29.944276 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.944179    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a2f424c-7201-4beb-acc4-4481306fb5a3-tls-cert\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:29.946642 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:29.946620    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a2f424c-7201-4beb-acc4-4481306fb5a3-tls-cert\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:30.000682 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.000662    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjv2f\" (UniqueName: \"kubernetes.io/projected/1a2f424c-7201-4beb-acc4-4481306fb5a3-kube-api-access-bjv2f\") pod \"authorino-784fb86d65-6kg8v\" (UID: \"1a2f424c-7201-4beb-acc4-4481306fb5a3\") " pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:30.082668 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.082584    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-784fb86d65-6kg8v"
Nov 03 20:40:30.449993 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.447328    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-784fb86d65-6kg8v"]
Nov 03 20:40:30.679239 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.679200    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" event={"ID":"c840bef8-ed6c-473b-99d3-ff601513d093","Type":"ContainerStarted","Data":"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"}
Nov 03 20:40:30.680374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.680350    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-784fb86d65-6kg8v" event={"ID":"1a2f424c-7201-4beb-acc4-4481306fb5a3","Type":"ContainerStarted","Data":"b19db9d0dd0ffbdc07f0ca5c095bf00c3d0b1b419242bc0ca2a5520b67fd8697"}
Nov 03 20:40:30.792848 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:30.792794    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podStartSLOduration=1.63987415 podStartE2EDuration="5.792781149s" totalImagesPullingTime="4.152906999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:30.791661869 +0000 UTC m=+655.718232107" watchObservedRunningTime="2026-11-03 20:40:30.792781149 +0000 UTC m=+655.719351443"
Nov 03 20:40:31.370374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:31.370339    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:31.371853 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:31.371826    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:31.371980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:31.371889    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:32.368510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.368419    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:32.369069 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.369019    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:32.697538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.697437    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-784fb86d65-6kg8v" event={"ID":"1a2f424c-7201-4beb-acc4-4481306fb5a3","Type":"ContainerStarted","Data":"d1bb7a5e92d7d918b78f5e4fd37f9649e3b7795fba7c9c738f51bf5b7215e790"}
Nov 03 20:40:32.812154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.812100    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-784fb86d65-6kg8v" podStartSLOduration=2.170307826 podStartE2EDuration="3.812086068s" totalImagesPullingTime="1.641778242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:32.811438624 +0000 UTC m=+657.738008855" watchObservedRunningTime="2026-11-03 20:40:32.812086068 +0000 UTC m=+657.738656305"
Nov 03 20:40:32.873854 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.873816    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:40:32.874323 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:32.874273    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" podUID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" containerName="authorino" containerID="cri-o://fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c" gracePeriod=30
Nov 03 20:40:33.178337 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.178312    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:40:33.294229 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.294200    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\" (UniqueName: \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\") pod \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") "
Nov 03 20:40:33.294425 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.294264    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\" (UniqueName: \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\") pod \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\" (UID: \"94ff5caf-b24d-4bd4-836b-f370e05b9c95\") "
Nov 03 20:40:33.296576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.296541    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q" pod "94ff5caf-b24d-4bd4-836b-f370e05b9c95" (UID: "94ff5caf-b24d-4bd4-836b-f370e05b9c95"). InnerVolumeSpecName "kube-api-access-plw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:33.306470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.306440    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert" pod "94ff5caf-b24d-4bd4-836b-f370e05b9c95" (UID: "94ff5caf-b24d-4bd4-836b-f370e05b9c95"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:33.370432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.370389    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:33.370805 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.370453    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:33.395451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.395422    4274 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94ff5caf-b24d-4bd4-836b-f370e05b9c95-tls-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:33.395451 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.395447    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plw7q\" (UniqueName: \"kubernetes.io/projected/94ff5caf-b24d-4bd4-836b-f370e05b9c95-kube-api-access-plw7q\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:33.703920 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.703823    4274 generic.go:356] "Generic (PLEG): container finished" podID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" containerID="fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c" exitCode=0
Nov 03 20:40:33.703920 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.703860    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687fd5c7fd-gxmns"
Nov 03 20:40:33.703920 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.703905    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" event={"ID":"94ff5caf-b24d-4bd4-836b-f370e05b9c95","Type":"ContainerDied","Data":"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"}
Nov 03 20:40:33.704217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.703952    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687fd5c7fd-gxmns" event={"ID":"94ff5caf-b24d-4bd4-836b-f370e05b9c95","Type":"ContainerDied","Data":"8ef3e540650297e8691ecbd46e8bfd0950d3d5a3b4b0518bbb139315c43812a5"}
Nov 03 20:40:33.704217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.703973    4274 scope.go:122] "RemoveContainer" containerID="fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"
Nov 03 20:40:33.714302 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.714281    4274 scope.go:122] "RemoveContainer" containerID="fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"
Nov 03 20:40:33.714629 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:40:33.714600    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c\": container with ID starting with fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c not found: ID does not exist" containerID="fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"
Nov 03 20:40:33.714688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.714643    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c"} err="failed to get container status \"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c\": rpc error: code = NotFound desc = could not find container \"fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c\": container with ID starting with fe5a382a5925cecdf7ffd423578733abcf9abf7e76f4855fb9538aa8c674374c not found: ID does not exist"
Nov 03 20:40:33.870063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.869750    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:40:33.875240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:33.875206    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687fd5c7fd-gxmns"]
Nov 03 20:40:34.370059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:34.369426    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:34.370059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:34.369509    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:35.213225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:35.213195    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" path="/var/lib/kubelet/pods/94ff5caf-b24d-4bd4-836b-f370e05b9c95/volumes"
Nov 03 20:40:35.370503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:35.370471    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:35.370802 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:35.370762    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:36.368366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:36.368331    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:36.368733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:36.368387    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:37.052256 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.052215    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"]
Nov 03 20:40:37.052600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.052586    4274 cpu_manager.go:414] "containerMap: removing container" podUID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" containerName="authorino"
Nov 03 20:40:37.052678 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.052600    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" containerName="authorino"
Nov 03 20:40:37.052678 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.052657    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ff5caf-b24d-4bd4-836b-f370e05b9c95" containerName="authorino"
Nov 03 20:40:37.056985 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.056962    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.105573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.105543    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 20:40:37.132101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.132058    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.132280 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.132246    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssdf\" (UniqueName: \"kubernetes.io/projected/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-kube-api-access-gssdf\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.151389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.151358    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nmvw4\""
Nov 03 20:40:37.152969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.152944    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"]
Nov 03 20:40:37.233354 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.233316    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.233517 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.233394    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssdf\" (UniqueName: \"kubernetes.io/projected/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-kube-api-access-gssdf\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.234080 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.234058    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.290133 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.290096    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssdf\" (UniqueName: \"kubernetes.io/projected/9b209916-fe0e-4b03-a767-e4f6e2e1a6c1-kube-api-access-gssdf\") pod \"payload-pre-processing-7b98bb678c-6b6ls\" (UID: \"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.369091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.368981    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:40:37.369091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.369027    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:40:37.374196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.374172    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:37.750834 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:37.750707    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"]
Nov 03 20:40:37.756857 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:40:37.756821    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b209916_fe0e_4b03_a767_e4f6e2e1a6c1.slice/crio-4b1adcd8b04743a71cf8cab47f46fb3a0f95f732c7c2c26fa617e1b46117ba89 WatchSource:0}: Error finding container 4b1adcd8b04743a71cf8cab47f46fb3a0f95f732c7c2c26fa617e1b46117ba89: Status 404 returned error can't find the container with id 4b1adcd8b04743a71cf8cab47f46fb3a0f95f732c7c2c26fa617e1b46117ba89
Nov 03 20:40:38.728857 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:38.728814    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls" event={"ID":"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1","Type":"ContainerStarted","Data":"4b1adcd8b04743a71cf8cab47f46fb3a0f95f732c7c2c26fa617e1b46117ba89"}
Nov 03 20:40:39.370194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:39.370157    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:40:39.370402 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:39.370238    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:40:40.368857 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.368176    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:40.412255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.412217    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:40:40.412420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.412283    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:40:40.740148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.740064    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls" event={"ID":"9b209916-fe0e-4b03-a767-e4f6e2e1a6c1","Type":"ContainerStarted","Data":"5607e1a02a188e5196cf4374fed6fad3795274d25c13d2098d7a258d9c3d8bb0"}
Nov 03 20:40:40.740148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.740135    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:40.849523 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.849471    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls" podStartSLOduration=1.098576566 podStartE2EDuration="3.849456559s" totalImagesPullingTime="2.750879993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:40.847605656 +0000 UTC m=+665.774175893" watchObservedRunningTime="2026-11-03 20:40:40.849456559 +0000 UTC m=+665.776026795"
Nov 03 20:40:40.893353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.893323    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:40.894164 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:40.894143    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:40:44.842413 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.842381    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"]
Nov 03 20:40:44.846058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.846024    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:44.897444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.897422    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 20:40:44.906999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.906968    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-metrics-certs\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:44.907118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.907019    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-kube-api-access-c4l4p\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:44.943482 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.943463    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:40:44.943573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.943467    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:40:44.943706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.943690    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-q9pxf\""
Nov 03 20:40:44.943785 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.943768    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 20:40:44.945248 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:44.945229    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"]
Nov 03 20:40:45.007512 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.007464    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-metrics-certs\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:45.007512 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.007495    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-kube-api-access-c4l4p\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:45.009775 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.009757    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-metrics-certs\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:45.064609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.064581    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4l4p\" (UniqueName: \"kubernetes.io/projected/9e72b48d-0d33-4e96-b726-2bc9ec75a7fc-kube-api-access-c4l4p\") pod \"ai-gateway-controller-658cf6f9cb-l4n4p\" (UID: \"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc\") " pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:45.158010 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.157960    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:45.518431 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.518358    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"]
Nov 03 20:40:45.520623 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:40:45.520590    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e72b48d_0d33_4e96_b726_2bc9ec75a7fc.slice/crio-f2e5ecd28f50346346330774c90a0a94ac2c41d6b8385773e2984eced233a09d WatchSource:0}: Error finding container f2e5ecd28f50346346330774c90a0a94ac2c41d6b8385773e2984eced233a09d: Status 404 returned error can't find the container with id f2e5ecd28f50346346330774c90a0a94ac2c41d6b8385773e2984eced233a09d
Nov 03 20:40:45.760730 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:45.760693    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p" event={"ID":"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc","Type":"ContainerStarted","Data":"f2e5ecd28f50346346330774c90a0a94ac2c41d6b8385773e2984eced233a09d"}
Nov 03 20:40:47.770924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:47.770889    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p" event={"ID":"9e72b48d-0d33-4e96-b726-2bc9ec75a7fc","Type":"ContainerStarted","Data":"d0453479ec6fed33d635a48d2ff8cae5a6237a264a7ba8e235ec0b6777c38a23"}
Nov 03 20:40:47.771389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:47.770990    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:40:47.879227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:47.879180    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p" podStartSLOduration=2.036647469 podStartE2EDuration="3.879168299s" totalImagesPullingTime="1.84252083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:47.878450546 +0000 UTC m=+672.805020784" watchObservedRunningTime="2026-11-03 20:40:47.879168299 +0000 UTC m=+672.805738536"
Nov 03 20:40:51.746587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:51.746560    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6b6ls"
Nov 03 20:40:57.364994 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:40:57.364959    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:40:58.777166 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:40:58.777136    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-658cf6f9cb-l4n4p"
Nov 03 20:41:27.364647 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:41:27.364612    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:41:51.159395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.159361    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 20:41:51.162819 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.162800    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.215257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.215229    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1e6fe1-openshift-default-dockercfg-gpj7x\""
Nov 03 20:41:51.263508 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.263478    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 20:41:51.272862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.272829    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.272983 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.272875    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.272983 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.272960    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273090 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.272986    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273090 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.273051    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.273139    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.273166    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.273189    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbm5\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.273312 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.273291    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374152 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374125    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374292 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374161    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374292 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374180    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374292 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374195    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbm5\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374292 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374242    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374292 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374282    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374539 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374303    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374539 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374344    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374539 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374369    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374568    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374699    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374766    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374780    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.374971 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.374954    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.376637 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.376617    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.376944 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.376926    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.431544 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.431481    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.432428 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.432405    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbm5\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.467644 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.467621    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 20:41:51.472163 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.472148    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.475927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.475912    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:51.534394 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.534350    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88228a-openshift-default-dockercfg-tb65c\""
Nov 03 20:41:51.535478 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.535450    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 20:41:51.575865 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.575828    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffpc\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576011 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.575877    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576069    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576119    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576214    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576258    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576287    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.576417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.576394    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677161    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffpc\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677221    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677259    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677279    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677316    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677351    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677390    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677420    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677467    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677786 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677754    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677785    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.677908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.677881    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.678076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.678056    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.678332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.678314    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.679644 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.679626    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.679780 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.679764    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.734631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.734557    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.734770 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.734702    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffpc\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.785587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.785561    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:51.892876 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.892853    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 20:41:51.895588 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:41:51.895551    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158e614f_f439_4a57_8c6e_a8d9ba3e1778.slice/crio-71c47bffeab99f5699625c76c96b800674a9dfcaa3198e7103b7f8dc3ea86da1 WatchSource:0}: Error finding container 71c47bffeab99f5699625c76c96b800674a9dfcaa3198e7103b7f8dc3ea86da1: Status 404 returned error can't find the container with id 71c47bffeab99f5699625c76c96b800674a9dfcaa3198e7103b7f8dc3ea86da1
Nov 03 20:41:51.897754 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.897724    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:51.897838 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.897784    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:51.897838 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:51.897811    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:52.041399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.041351    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerStarted","Data":"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"}
Nov 03 20:41:52.041577 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.041414    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerStarted","Data":"71c47bffeab99f5699625c76c96b800674a9dfcaa3198e7103b7f8dc3ea86da1"}
Nov 03 20:41:52.161098 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.161018    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podStartSLOduration=1.160997321 podStartE2EDuration="1.160997321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:52.158792147 +0000 UTC m=+737.085362405" watchObservedRunningTime="2026-11-03 20:41:52.160997321 +0000 UTC m=+737.087567557"
Nov 03 20:41:52.165311 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.165276    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 20:41:52.166622 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:41:52.166590    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802b6f5e_10d7_46ce_8c44_3eb446983e13.slice/crio-7a5c3ea4a2bd8efde7f43f7b526b9183f7504b2dcf148db974ce626bca8258f1 WatchSource:0}: Error finding container 7a5c3ea4a2bd8efde7f43f7b526b9183f7504b2dcf148db974ce626bca8258f1: Status 404 returned error can't find the container with id 7a5c3ea4a2bd8efde7f43f7b526b9183f7504b2dcf148db974ce626bca8258f1
Nov 03 20:41:52.169808 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.169643    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:52.169808 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.169697    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:52.169808 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.169723    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:52.395851 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.395773    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 20:41:52.399914 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.399890    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.451591 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.451559    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-504eb0-openshift-default-dockercfg-4ckhb\""
Nov 03 20:41:52.458379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.458350    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 20:41:52.476616 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.476592    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:52.482528 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.482508    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:52.486084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486059    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486164 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486116    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486168    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486273 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486250    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486302    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc6s\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486324    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486344    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486467 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486448    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.486536 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.486519    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.587986 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.587951    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.587986 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.587995    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588010    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588043    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588074    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc6s\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588091    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588116    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588143    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588198    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588421    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588451    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588856    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588954 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588940    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.588954 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.588940    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.590652 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.590629    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.590824 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.590802    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.645699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.645665    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.645844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.645786    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc6s\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.712899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.712820    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:52.716211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.716179    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 20:41:52.719598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.719576    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.777623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.777594    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2a5ec9-openshift-default-dockercfg-kln2c\""
Nov 03 20:41:52.790073 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790016    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790082    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790170    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790192    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790212    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790232    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790370 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790338    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790405 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790382    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwss\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.790479 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.790454    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.829454 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.829421    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 20:41:52.891945 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.891909    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.891977    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892002    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892027    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892064    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892194    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892274    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwss\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892327    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892523 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892441    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892523 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892450    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892627 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892526    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892685 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892629    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892781    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.892869 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.892824    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.894263 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.894244    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.894712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.894693    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.977992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.977925    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:52.978139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:52.978010    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwss\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:53.034662 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.034632    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:53.047901 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.047871    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerStarted","Data":"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"}
Nov 03 20:41:53.048014 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.047934    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerStarted","Data":"7a5c3ea4a2bd8efde7f43f7b526b9183f7504b2dcf148db974ce626bca8258f1"}
Nov 03 20:41:53.048167 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.048149    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:53.049105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.049090    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:41:53.114331 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.114307    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 20:41:53.115145 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:41:53.115120    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0b00b5_fa79_47fd_aee5_710fcb1304ff.slice/crio-2a65bd0ddd0a21e07a51600dd9ee75222ca3cc24df8ff41d3caaa74a1dce307c WatchSource:0}: Error finding container 2a65bd0ddd0a21e07a51600dd9ee75222ca3cc24df8ff41d3caaa74a1dce307c: Status 404 returned error can't find the container with id 2a65bd0ddd0a21e07a51600dd9ee75222ca3cc24df8ff41d3caaa74a1dce307c
Nov 03 20:41:53.117733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.117704    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.117801 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.117762    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.117801 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.117790    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.165588 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.165550    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podStartSLOduration=2.165536857 podStartE2EDuration="2.165536857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:53.164285913 +0000 UTC m=+738.090856149" watchObservedRunningTime="2026-11-03 20:41:53.165536857 +0000 UTC m=+738.092107091"
Nov 03 20:41:53.404203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.404167    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 20:41:53.410522 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:41:53.410220    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb90bc3_66ab_44d3_81ff_8b95435a177c.slice/crio-684a38dbee8e23a5bdca916c85c69c1ada615b7e4185eb270cb06643a73c898a WatchSource:0}: Error finding container 684a38dbee8e23a5bdca916c85c69c1ada615b7e4185eb270cb06643a73c898a: Status 404 returned error can't find the container with id 684a38dbee8e23a5bdca916c85c69c1ada615b7e4185eb270cb06643a73c898a
Nov 03 20:41:53.412740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.412701    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.412838 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.412792    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.412895 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.412862    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:41:53.786739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.786706    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:53.791477 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:53.791454    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:54.053274 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.053175    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerStarted","Data":"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"}
Nov 03 20:41:54.053274 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.053226    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerStarted","Data":"2a65bd0ddd0a21e07a51600dd9ee75222ca3cc24df8ff41d3caaa74a1dce307c"}
Nov 03 20:41:54.054781 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.054746    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerStarted","Data":"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"}
Nov 03 20:41:54.054898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.054797    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerStarted","Data":"684a38dbee8e23a5bdca916c85c69c1ada615b7e4185eb270cb06643a73c898a"}
Nov 03 20:41:54.054898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.054822    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:54.055732 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.055711    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:41:54.167178 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.167127    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podStartSLOduration=2.167114296 podStartE2EDuration="2.167114296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:54.165792548 +0000 UTC m=+739.092362785" watchObservedRunningTime="2026-11-03 20:41:54.167114296 +0000 UTC m=+739.093684607"
Nov 03 20:41:54.396853 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.396757    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podStartSLOduration=2.396742314 podStartE2EDuration="2.396742314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:54.395890547 +0000 UTC m=+739.322460784" watchObservedRunningTime="2026-11-03 20:41:54.396742314 +0000 UTC m=+739.323312551"
Nov 03 20:41:54.713149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.713066    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:54.718080 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:54.718057    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:55.035617 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.035584    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:55.040624 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.040600    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:55.064665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.063998    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:55.064665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.064136    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:41:55.064665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.064167    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:55.065298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:55.065279    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:41:57.364085 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:41:57.364048    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:41:59.689568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.689538    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 20:41:59.694357 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.694337    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:41:59.791858 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.791798    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1e6fe1\""
Nov 03 20:41:59.792068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.791902    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1e6fe1-dockercfg-q9fmv\""
Nov 03 20:41:59.793843 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.793817    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 20:41:59.810191 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.810165    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 20:41:59.813818 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.813800    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:41:59.865425 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.865397    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:41:59.865526 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.865457    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7kj\" (UniqueName: \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:41:59.877512 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.877491    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 20:41:59.966468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.966395    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:41:59.966468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.966455    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:41:59.966631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.966487    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7kj\" (UniqueName: \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:41:59.966631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.966529    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7qc\" (UniqueName: \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:41:59.967154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:41:59.967134    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:42:00.024005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.023979    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7kj\" (UniqueName: \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\") pod \"payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:42:00.067494 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.067465    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:00.067605 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.067517    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7qc\" (UniqueName: \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:00.068205 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.068187    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:00.124185 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.124158    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7qc\" (UniqueName: \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\") pod \"payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:00.125993 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.125974    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:00.306464 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.306436    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:42:00.489558 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.489535    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 20:42:00.491554 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:00.491525    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce54e08b_44d3_408b_810f_c4d555985350.slice/crio-b679b4f4d2dad8ee83bdf7a5d47f9cf0f3c0f442889b136597c5186905f4737a WatchSource:0}: Error finding container b679b4f4d2dad8ee83bdf7a5d47f9cf0f3c0f442889b136597c5186905f4737a: Status 404 returned error can't find the container with id b679b4f4d2dad8ee83bdf7a5d47f9cf0f3c0f442889b136597c5186905f4737a
Nov 03 20:42:00.670883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:00.670860    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 20:42:00.672521 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:00.672497    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6273b213_7f67_433c_97d7_8178be69f048.slice/crio-5ce783e7a4b5e207edac2801f0303f9dfcce97dd050b85969ca468860c1f83e8 WatchSource:0}: Error finding container 5ce783e7a4b5e207edac2801f0303f9dfcce97dd050b85969ca468860c1f83e8: Status 404 returned error can't find the container with id 5ce783e7a4b5e207edac2801f0303f9dfcce97dd050b85969ca468860c1f83e8
Nov 03 20:42:01.089110 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.089068    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" event={"ID":"6273b213-7f67-433c-97d7-8178be69f048","Type":"ContainerStarted","Data":"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"}
Nov 03 20:42:01.089573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.089130    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" event={"ID":"6273b213-7f67-433c-97d7-8178be69f048","Type":"ContainerStarted","Data":"5ce783e7a4b5e207edac2801f0303f9dfcce97dd050b85969ca468860c1f83e8"}
Nov 03 20:42:01.089573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.089175    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:42:01.090552 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.090523    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" event={"ID":"ce54e08b-44d3-408b-810f-c4d555985350","Type":"ContainerStarted","Data":"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"}
Nov 03 20:42:01.090694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.090573    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" event={"ID":"ce54e08b-44d3-408b-810f-c4d555985350","Type":"ContainerStarted","Data":"b679b4f4d2dad8ee83bdf7a5d47f9cf0f3c0f442889b136597c5186905f4737a"}
Nov 03 20:42:01.090694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.090606    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:01.185429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.185393    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 20:42:01.189004 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.188988    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.198931 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.198895    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" podStartSLOduration=2.198883768 podStartE2EDuration="2.198883768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:01.197380854 +0000 UTC m=+746.123951090" watchObservedRunningTime="2026-11-03 20:42:01.198883768 +0000 UTC m=+746.125454004"
Nov 03 20:42:01.237709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.237689    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-3b8feb-serving-cert\""
Nov 03 20:42:01.239237 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.239216    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 20:42:01.381204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.381125    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.381204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.381181    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f62q\" (UniqueName: \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.381204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.381202    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.381432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.381400    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.414350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.414275    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" podStartSLOduration=2.414255678 podStartE2EDuration="2.414255678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:01.412888716 +0000 UTC m=+746.339458953" watchObservedRunningTime="2026-11-03 20:42:01.414255678 +0000 UTC m=+746.340825914"
Nov 03 20:42:01.482684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.482643    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.482883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.482695    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f62q\" (UniqueName: \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.482883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.482724    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.482883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.482756    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.483487 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.483462    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.485199 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.485178    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.485266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.485210    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.540842 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.540808    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f62q\" (UniqueName: \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\") pod \"maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:01.802065 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:01.802015    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:02.215278 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:02.215251    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 20:42:02.216913 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:02.216883    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd915e890_e4be_485e_b610_5dba66745dad.slice/crio-2b2c806e65564cd2cc0305360164355fa6a2c93bb9e1230a55c5a3c23f09d32e WatchSource:0}: Error finding container 2b2c806e65564cd2cc0305360164355fa6a2c93bb9e1230a55c5a3c23f09d32e: Status 404 returned error can't find the container with id 2b2c806e65564cd2cc0305360164355fa6a2c93bb9e1230a55c5a3c23f09d32e
Nov 03 20:42:03.100912 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:03.100874    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" event={"ID":"d915e890-e4be-485e-b610-5dba66745dad","Type":"ContainerStarted","Data":"2b2c806e65564cd2cc0305360164355fa6a2c93bb9e1230a55c5a3c23f09d32e"}
Nov 03 20:42:04.106640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:04.106593    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" event={"ID":"d915e890-e4be-485e-b610-5dba66745dad","Type":"ContainerStarted","Data":"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"}
Nov 03 20:42:04.107167 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:04.106732    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:04.226753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:04.226689    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" podStartSLOduration=2.3199571629999998 podStartE2EDuration="3.226671771s" totalImagesPullingTime="906.714608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:04.225515386 +0000 UTC m=+749.152085624" watchObservedRunningTime="2026-11-03 20:42:04.226671771 +0000 UTC m=+749.153242008"
Nov 03 20:42:06.323876 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.323845    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 20:42:06.327733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.327709    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.384967 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.384944    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-88228a-serving-cert\""
Nov 03 20:42:06.427782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.427748    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.427924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.427904    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.427985 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.427936    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.428064 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.428023    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzptd\" (UniqueName: \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.430196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.430177    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 20:42:06.528771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.528734    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.529030 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.528902    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.529179 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:06.529158    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-serving-cert: secret "maas-api-e2e-worker-w3-88228a-serving-cert" not found
Nov 03 20:42:06.529293 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:06.529283    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls podName:78cc0ffc-1bf2-4901-81e6-9a80273f07c5 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:07.029259815 +0000 UTC m=+751.955830054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls") pod "maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" (UID: "78cc0ffc-1bf2-4901-81e6-9a80273f07c5") : secret "maas-api-e2e-worker-w3-88228a-serving-cert" not found
Nov 03 20:42:06.529788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.529760    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.529924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.529823    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.529989 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.529836    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzptd\" (UniqueName: \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.532649 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.532613    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:06.587575 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:06.587511    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzptd\" (UniqueName: \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:07.035027 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:07.034996    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:07.037356 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:07.037336    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:07.240513 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:07.240476    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:07.604338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:07.604310    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 20:42:07.605361 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:07.605331    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cc0ffc_1bf2_4901_81e6_9a80273f07c5.slice/crio-8ccf363877fb71730dc25a79c4b57670ccb7414804eaed6d56463fd41953382b WatchSource:0}: Error finding container 8ccf363877fb71730dc25a79c4b57670ccb7414804eaed6d56463fd41953382b: Status 404 returned error can't find the container with id 8ccf363877fb71730dc25a79c4b57670ccb7414804eaed6d56463fd41953382b
Nov 03 20:42:08.125501 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:08.125466    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" event={"ID":"78cc0ffc-1bf2-4901-81e6-9a80273f07c5","Type":"ContainerStarted","Data":"8ccf363877fb71730dc25a79c4b57670ccb7414804eaed6d56463fd41953382b"}
Nov 03 20:42:09.131200 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:09.131164    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" event={"ID":"78cc0ffc-1bf2-4901-81e6-9a80273f07c5","Type":"ContainerStarted","Data":"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"}
Nov 03 20:42:09.131671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:09.131275    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:09.241736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:09.241683    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" podStartSLOduration=2.247681783 podStartE2EDuration="3.241669975s" totalImagesPullingTime="993.988192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:09.241359508 +0000 UTC m=+754.167929747" watchObservedRunningTime="2026-11-03 20:42:09.241669975 +0000 UTC m=+754.168240225"
Nov 03 20:42:10.116769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:10.116732    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 20:42:12.099944 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:12.099914    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 20:42:12.100472 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:12.099998    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 20:42:14.815657 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.815627    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 20:42:14.819238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.819217    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:14.867873 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.867848    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-504eb0\""
Nov 03 20:42:14.911843 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.909614    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:14.911843 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.909901    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgppz\" (UniqueName: \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:14.913854 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.913831    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-504eb0-dockercfg-cvp8d\""
Nov 03 20:42:14.916253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:14.916233    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 20:42:15.011240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.011175    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgppz\" (UniqueName: \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:15.011240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.011235    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:15.012136 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.012117    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:15.068522 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.068463    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgppz\" (UniqueName: \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\") pod \"payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:15.131208 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.131188    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:15.142466 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.142446    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 20:42:15.496904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:15.496879    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 20:42:15.498521 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:15.498494    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb891c86c_18e7_4a20_9d8f_af05861c1599.slice/crio-476b65bbdfa54f176965a7c90fb89372a5e4fab00901cfbdcfa9fe852d615092 WatchSource:0}: Error finding container 476b65bbdfa54f176965a7c90fb89372a5e4fab00901cfbdcfa9fe852d615092: Status 404 returned error can't find the container with id 476b65bbdfa54f176965a7c90fb89372a5e4fab00901cfbdcfa9fe852d615092
Nov 03 20:42:16.167346 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:16.167313    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" event={"ID":"b891c86c-18e7-4a20-9d8f-af05861c1599","Type":"ContainerStarted","Data":"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"}
Nov 03 20:42:16.167773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:16.167397    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" event={"ID":"b891c86c-18e7-4a20-9d8f-af05861c1599","Type":"ContainerStarted","Data":"476b65bbdfa54f176965a7c90fb89372a5e4fab00901cfbdcfa9fe852d615092"}
Nov 03 20:42:16.167773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:16.167521    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:16.275797 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:16.275746    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" podStartSLOduration=2.275734309 podStartE2EDuration="2.275734309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:16.275144359 +0000 UTC m=+761.201714608" watchObservedRunningTime="2026-11-03 20:42:16.275734309 +0000 UTC m=+761.202304546"
Nov 03 20:42:20.599749 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.599710    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 20:42:20.603517 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.603497    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.660972 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.660937    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3d2008-serving-cert\""
Nov 03 20:42:20.692190 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.692158    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.692379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.692217    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.692379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.692356    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gkx\" (UniqueName: \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.692505 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.692394    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.708769 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.708747    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 20:42:20.793902 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.793872    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gkx\" (UniqueName: \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.794085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.793921    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.794085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.793975    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.794085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.794021    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.794211 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:20.794189    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-serving-cert: secret "maas-api-e2e-worker-w4-3d2008-serving-cert" not found
Nov 03 20:42:20.794284 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:20.794272    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls podName:dc238b02-aaa7-418b-990b-ed194b40c725 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:21.294249591 +0000 UTC m=+766.220819805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls") pod "maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" (UID: "dc238b02-aaa7-418b-990b-ed194b40c725") : secret "maas-api-e2e-worker-w4-3d2008-serving-cert" not found
Nov 03 20:42:20.794725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.794707    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.796283 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.796262    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:20.877132 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:20.863778    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gkx\" (UniqueName: \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:21.298477 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:21.298441    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:21.300772 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:21.300744    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:21.515869 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:21.515834    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:21.911195 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:21.911169    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 20:42:21.913880 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:21.913849    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc238b02_aaa7_418b_990b_ed194b40c725.slice/crio-7412e12ec2ce8fc8265399634b77121d1f352e68e1c8c8a61151f6be6b1af546 WatchSource:0}: Error finding container 7412e12ec2ce8fc8265399634b77121d1f352e68e1c8c8a61151f6be6b1af546: Status 404 returned error can't find the container with id 7412e12ec2ce8fc8265399634b77121d1f352e68e1c8c8a61151f6be6b1af546
Nov 03 20:42:22.201200 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:22.201075    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" event={"ID":"dc238b02-aaa7-418b-990b-ed194b40c725","Type":"ContainerStarted","Data":"7412e12ec2ce8fc8265399634b77121d1f352e68e1c8c8a61151f6be6b1af546"}
Nov 03 20:42:23.211503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:23.211430    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:23.211503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:23.211473    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" event={"ID":"dc238b02-aaa7-418b-990b-ed194b40c725","Type":"ContainerStarted","Data":"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"}
Nov 03 20:42:23.324973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:23.324916    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" podStartSLOduration=2.409215054 podStartE2EDuration="3.324902572s" totalImagesPullingTime="915.687518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:23.324076576 +0000 UTC m=+768.250646814" watchObservedRunningTime="2026-11-03 20:42:23.324902572 +0000 UTC m=+768.251472809"
Nov 03 20:42:25.097178 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:25.097143    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:42:25.097684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:25.097203    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:27.178058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.177970    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 20:42:27.228660 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.228587    4274 generic.go:356] "Generic (PLEG): container finished" podID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerID="dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe" exitCode=0
Nov 03 20:42:27.228660 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.228651    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerDied","Data":"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"}
Nov 03 20:42:27.229000 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.228985    4274 scope.go:122] "RemoveContainer" containerID="dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"
Nov 03 20:42:27.230296 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.230271    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:27.230363 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.230335    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:27.230402 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:27.230368    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:27.403951 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:27.403915    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:42:28.235211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:28.235171    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerStarted","Data":"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"}
Nov 03 20:42:28.794063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:28.793596    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:42:28.794063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:28.793733    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:42:28.794063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:28.793778    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 20:42:29.218255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:29.218180    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 20:42:29.792060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:29.791462    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:42:29.792060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:29.791542    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 20:42:30.788439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:30.788399    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:42:30.788764 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:30.788721    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 20:42:31.786143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:31.786105    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:42:31.786583 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:31.786170    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 20:42:32.786326 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:32.786288    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:42:32.786730 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:32.786377    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 20:42:33.789865 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:33.789834    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:42:33.790253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:33.790232    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:42:33.790961 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:33.790941    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 20:42:34.869424 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.869390    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 20:42:34.873694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.873653    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:34.958383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.958356    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:34.958486 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.958403    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:34.958486 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.958437    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lnb\" (UniqueName: \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:34.958571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.958551    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:34.970746 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.970719    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-504eb0-serving-cert\""
Nov 03 20:42:34.990005 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:34.989972    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 20:42:35.059343 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.059311    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lnb\" (UniqueName: \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.059506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.059378    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.059506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.059455    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.059622 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.059504    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.059682 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:35.059623    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-serving-cert: secret "maas-api-e2e-worker-w6-504eb0-serving-cert" not found
Nov 03 20:42:35.059736 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:35.059708    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls podName:31955084-06da-41d1-9adb-5a44bc622645 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:35.559684733 +0000 UTC m=+780.486254962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls") pod "maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" (UID: "31955084-06da-41d1-9adb-5a44bc622645") : secret "maas-api-e2e-worker-w6-504eb0-serving-cert" not found
Nov 03 20:42:35.060180 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.060154    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.062260 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.062232    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.131212 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.131138    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lnb\" (UniqueName: \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.283830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.283795    4274 generic.go:356] "Generic (PLEG): container finished" podID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerID="943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3" exitCode=0
Nov 03 20:42:35.283980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.283862    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerDied","Data":"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"}
Nov 03 20:42:35.284321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.284306    4274 scope.go:122] "RemoveContainer" containerID="943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"
Nov 03 20:42:35.285416 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.285392    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:35.285485 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.285454    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:35.285519 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.285484    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:35.565144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.565113    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.567406 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.567383    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") pod \"maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:35.793478 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:35.793435    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:36.194007 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.193982    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 20:42:36.195924 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:42:36.195835    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31955084_06da_41d1_9adb_5a44bc622645.slice/crio-95374b3ea9eec29cad199be96f541e1f29bfd18fcdf64312f2a133468a9500ea WatchSource:0}: Error finding container 95374b3ea9eec29cad199be96f541e1f29bfd18fcdf64312f2a133468a9500ea: Status 404 returned error can't find the container with id 95374b3ea9eec29cad199be96f541e1f29bfd18fcdf64312f2a133468a9500ea
Nov 03 20:42:36.289373 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.289341    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" event={"ID":"31955084-06da-41d1-9adb-5a44bc622645","Type":"ContainerStarted","Data":"95374b3ea9eec29cad199be96f541e1f29bfd18fcdf64312f2a133468a9500ea"}
Nov 03 20:42:36.291240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.291213    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerStarted","Data":"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"}
Nov 03 20:42:36.476346 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.476252    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:42:36.477339 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.477317    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:42:36.477393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:36.477366    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:42:37.297968 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:37.297928    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" event={"ID":"31955084-06da-41d1-9adb-5a44bc622645","Type":"ContainerStarted","Data":"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"}
Nov 03 20:42:37.298373 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:37.298114    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:37.432970 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:37.432866    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" podStartSLOduration=2.528795111 podStartE2EDuration="3.432851459s" totalImagesPullingTime="904.056348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:37.431386645 +0000 UTC m=+782.357956884" watchObservedRunningTime="2026-11-03 20:42:37.432851459 +0000 UTC m=+782.359421695"
Nov 03 20:42:37.476812 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:37.476780    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:42:37.476969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:37.476837    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:42:38.052060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:38.050134    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:42:38.476777 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:38.476681    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:42:38.476777 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:38.476761    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:42:39.476601 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:39.476565    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:42:39.476787 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:39.476618    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:42:40.476566 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:40.476528    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:42:40.476739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:40.476588    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:42:41.480865 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:41.480834    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:42:41.481706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:41.481687    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 20:42:43.308825 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:43.308794    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 20:42:55.063548 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.063511    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:42:55.064001 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.063565    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:42:55.430074 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.430017    4274 generic.go:356] "Generic (PLEG): container finished" podID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerID="4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4" exitCode=0
Nov 03 20:42:55.430237 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.430076    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerDied","Data":"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"}
Nov 03 20:42:55.430454 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.430437    4274 scope.go:122] "RemoveContainer" containerID="4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"
Nov 03 20:42:55.431664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.431636    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:55.431751 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.431702    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:55.431751 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.431735    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:55.754114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.752909    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:53300->10.133.0.38:15021: read: connection reset by peer" start-of-body=
Nov 03 20:42:55.754114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:55.752972    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:53300->10.133.0.38:15021: read: connection reset by peer"
Nov 03 20:42:56.436341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.436307    4274 generic.go:356] "Generic (PLEG): container finished" podID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerID="a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce" exitCode=0
Nov 03 20:42:56.436830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.436384    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerDied","Data":"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"}
Nov 03 20:42:56.436901 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.436845    4274 scope.go:122] "RemoveContainer" containerID="a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"
Nov 03 20:42:56.438277 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.438256    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerStarted","Data":"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"}
Nov 03 20:42:56.438381 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.438283    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:56.438381 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.438351    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:56.438448 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:56.438383    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:42:57.035897 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.035854    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:42:57.036150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.036126    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:42:57.036242 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.036178    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:42:57.419081 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:42:57.418989    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:42:57.481631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.481592    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerStarted","Data":"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"}
Nov 03 20:42:57.713487 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.713405    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:42:57.713633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.713483    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:42:57.713741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:57.713721    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:42:58.036027 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:58.036000    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:42:58.036197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:58.036069    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:42:58.718783 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:58.713414    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:42:58.718783 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:58.713465    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:42:59.035719 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:59.035683    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:42:59.035908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:59.035736    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:42:59.713315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:59.713283    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:42:59.713492 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:42:59.713335    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:43:00.035733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:00.035701    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:43:00.036161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:00.035755    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:43:00.713950 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:00.713914    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:43:00.714149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:00.713966    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:43:01.035738 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.035705    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:43:01.036119 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.035757    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:43:01.314336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.314267    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 20:43:01.318145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.318114    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.358907    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.358963    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.359009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmxp\" (UniqueName: \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.359085    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.359160    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.397303 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.359192    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.439925 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.439880    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"kube-root-ca.crt\""
Nov 03 20:43:01.440082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.439965    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"openshift-service-ca.crt\""
Nov 03 20:43:01.440082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.439996    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"simulator-w3-88228a-kserve-self-signed-certs\""
Nov 03 20:43:01.440744 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.440720    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"default-dockercfg-8xb4k\""
Nov 03 20:43:01.453058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.445097    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 20:43:01.493061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.464086    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.497058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.493345    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.501219 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.501188    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.507173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.507142    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.507335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.471528    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.507515 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.507493    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmxp\" (UniqueName: \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.507696 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.507679    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.508242 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.498440    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.509127 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.509100    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.512286 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.512264    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.512682 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.512660    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.582800 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.582725    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmxp\" (UniqueName: \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\") pod \"simulator-w3-88228a-kserve-77fbcc8779-gfztm\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") " pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.630149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.630101    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:01.713224 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.713198    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:43:01.713350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:01.713249    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:43:02.026188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.026156    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 20:43:02.028796 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:43:02.028768    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4fadca4_3d39_460b_9ee4_01aff0c65391.slice/crio-b4b6f1f43769d9303ed842ca517df2feddb4dc19479b8e1a7d55e70ad126d309 WatchSource:0}: Error finding container b4b6f1f43769d9303ed842ca517df2feddb4dc19479b8e1a7d55e70ad126d309: Status 404 returned error can't find the container with id b4b6f1f43769d9303ed842ca517df2feddb4dc19479b8e1a7d55e70ad126d309
Nov 03 20:43:02.040819 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.040791    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:43:02.041213 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.041185    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:43:02.041899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.041876    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 20:43:02.524496 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.524466    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" event={"ID":"d4fadca4-3d39-460b-9ee4-01aff0c65391","Type":"ContainerStarted","Data":"91ca739c5612bbfaa80849d988c710f2bc795af7429a0a957c8deff354d192b7"}
Nov 03 20:43:02.524667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.524510    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" event={"ID":"d4fadca4-3d39-460b-9ee4-01aff0c65391","Type":"ContainerStarted","Data":"b4b6f1f43769d9303ed842ca517df2feddb4dc19479b8e1a7d55e70ad126d309"}
Nov 03 20:43:02.630694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.630645    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:02.717395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.717365    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:43:02.717680 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.717659    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:43:02.718386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.718369    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 20:43:02.832217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:02.832122    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podStartSLOduration=1.832105806 podStartE2EDuration="1.832105806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:02.63359826 +0000 UTC m=+807.560168498" watchObservedRunningTime="2026-11-03 20:43:02.832105806 +0000 UTC m=+807.758676042"
Nov 03 20:43:11.630432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:11.630396    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:11.642770 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:11.642746    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:12.585601 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:12.585573    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:43:24.451723 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.451688    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 20:43:24.455400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.455378    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.504380 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.504355    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"openshift-service-ca.crt\""
Nov 03 20:43:24.504380 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.504359    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"kube-root-ca.crt\""
Nov 03 20:43:24.532193 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532164    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54kf\" (UniqueName: \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.532306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532253    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.532306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532292    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.532413 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532312    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.532413 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532327    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.532515 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.532419    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.552472 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.552442    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 20:43:24.585390 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.585369    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"default-dockercfg-tv7ld\""
Nov 03 20:43:24.585487 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.585447    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"premium-w2-3b8feb-kserve-self-signed-certs\""
Nov 03 20:43:24.633137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633107    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54kf\" (UniqueName: \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.633330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633165    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.633330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633207    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.633330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633233    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.633330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633253    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.633330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633298    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.634313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.633985    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.634313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.634069    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.634878 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.634855    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.638087 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.638060    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.638305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.638282    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.690444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.690419    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54kf\" (UniqueName: \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\") pod \"premium-w2-3b8feb-kserve-f465d7b96-tz2kn\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") " pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:24.767563 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:24.767532    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:25.133461 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:25.133427    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 20:43:25.134847 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:43:25.134812    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1561b796_6182_4059_a706_9b53b22506b0.slice/crio-839139347f7f8d9f640ea8a4a64b9e5da9708debbd13d9fa9feb4c3418decb62 WatchSource:0}: Error finding container 839139347f7f8d9f640ea8a4a64b9e5da9708debbd13d9fa9feb4c3418decb62: Status 404 returned error can't find the container with id 839139347f7f8d9f640ea8a4a64b9e5da9708debbd13d9fa9feb4c3418decb62
Nov 03 20:43:25.637544 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:25.637514    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" event={"ID":"1561b796-6182-4059-a706-9b53b22506b0","Type":"ContainerStarted","Data":"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"}
Nov 03 20:43:25.637949 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:25.637552    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" event={"ID":"1561b796-6182-4059-a706-9b53b22506b0","Type":"ContainerStarted","Data":"839139347f7f8d9f640ea8a4a64b9e5da9708debbd13d9fa9feb4c3418decb62"}
Nov 03 20:43:25.747791 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:25.747745    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podStartSLOduration=1.747733 podStartE2EDuration="1.747733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:25.746164107 +0000 UTC m=+830.672734345" watchObservedRunningTime="2026-11-03 20:43:25.747733 +0000 UTC m=+830.674303237"
Nov 03 20:43:25.767747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:25.767721    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:27.364482 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:43:27.364446    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:43:34.768745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:34.768707    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:34.782149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:34.782127    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:35.344373 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.344340    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 20:43:35.349139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.349118    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.398370 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.398342    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"kube-root-ca.crt\""
Nov 03 20:43:35.401815 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.401793    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"openshift-service-ca.crt\""
Nov 03 20:43:35.443651 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.443629    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"simulator-w4-3d2008-kserve-self-signed-certs\""
Nov 03 20:43:35.443862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.443845    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"default-dockercfg-8nct2\""
Nov 03 20:43:35.446021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.446005    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 20:43:35.448473 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448457    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.448526 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448498    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.448570 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448556    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.448609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448581    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.448609 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448598    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.448701 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.448650    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4r5\" (UniqueName: \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.549964 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.549930    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.549975    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550000    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550021    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550070    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550105    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4r5\" (UniqueName: \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550407    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550566 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550549    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.550619 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.550566    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.552326 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.552292    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.552562 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.552544    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.606941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.606882    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4r5\" (UniqueName: \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\") pod \"simulator-w4-3d2008-kserve-679447b6fb-c6jlz\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") " pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.660935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.660898    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:35.697517 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:35.697486    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:43:36.030870 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:36.030846    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 20:43:36.032311 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:43:36.032276    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9734fd48_ac42_436e_8e0a_9e8b3895ddc5.slice/crio-c1a94fa8afd624edfba94a22c637b406106426e299a09f7ec88a74b5f1c56df5 WatchSource:0}: Error finding container c1a94fa8afd624edfba94a22c637b406106426e299a09f7ec88a74b5f1c56df5: Status 404 returned error can't find the container with id c1a94fa8afd624edfba94a22c637b406106426e299a09f7ec88a74b5f1c56df5
Nov 03 20:43:36.692487 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:36.692450    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" event={"ID":"9734fd48-ac42-436e-8e0a-9e8b3895ddc5","Type":"ContainerStarted","Data":"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"}
Nov 03 20:43:36.692655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:36.692501    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" event={"ID":"9734fd48-ac42-436e-8e0a-9e8b3895ddc5","Type":"ContainerStarted","Data":"c1a94fa8afd624edfba94a22c637b406106426e299a09f7ec88a74b5f1c56df5"}
Nov 03 20:43:36.835077 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:36.835005    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podStartSLOduration=1.834993482 podStartE2EDuration="1.834993482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:36.833902521 +0000 UTC m=+841.760472759" watchObservedRunningTime="2026-11-03 20:43:36.834993482 +0000 UTC m=+841.761563718"
Nov 03 20:43:37.661526 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:37.661484    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:45.661395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:45.661325    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:45.673568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:45.673543    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:45.748389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:43:45.748361    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:43:57.364067 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:43:57.364019    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:44:02.817331 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:02.817292    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal" podUID="1c4b446c071455ed0e37e47646678b6e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:44:04.983237 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:04.983204    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 20:44:04.986973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:04.986954    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.036060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.036003    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"openshift-service-ca.crt\""
Nov 03 20:44:05.082194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.082156    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"kube-root-ca.crt\""
Nov 03 20:44:05.082464 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.082271    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"simulator-w6-504eb0-kserve-self-signed-certs\""
Nov 03 20:44:05.082464 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.082401    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"default-dockercfg-56s89\""
Nov 03 20:44:05.084696 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.084669    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 20:44:05.137829 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.137792    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.138018 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.137841    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.138018 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.137943    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.138018 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.138009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.138214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.138099    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7z7\" (UniqueName: \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.138214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.138178    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238672    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7z7\" (UniqueName: \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238739    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238788    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238811    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238845    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.238940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.238911    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.239135 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.239108    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.239232 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.239202    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.239232 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.239214    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.241051 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.241016    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.241437 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.241417    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.296509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.296479    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7z7\" (UniqueName: \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\") pod \"simulator-w6-504eb0-kserve-545988648f-tt95s\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") " pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.299351 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.299326    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:05.678177 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.678149    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 20:44:05.680009 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:44:05.679980    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3627eb01_f4b2_4012_8cc9_a6f0383bc610.slice/crio-04d911cfce20483c2d4ed50eb6bfb0fb9ffbb552a2e090be9754ab9204715e95 WatchSource:0}: Error finding container 04d911cfce20483c2d4ed50eb6bfb0fb9ffbb552a2e090be9754ab9204715e95: Status 404 returned error can't find the container with id 04d911cfce20483c2d4ed50eb6bfb0fb9ffbb552a2e090be9754ab9204715e95
Nov 03 20:44:05.834348 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.834314    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" event={"ID":"3627eb01-f4b2-4012-8cc9-a6f0383bc610","Type":"ContainerStarted","Data":"eab9a9e78dc01a11f11f5bfc95844a621a2c61759b8f65d89ead8ef36c1c4e64"}
Nov 03 20:44:05.834530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.834364    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" event={"ID":"3627eb01-f4b2-4012-8cc9-a6f0383bc610","Type":"ContainerStarted","Data":"04d911cfce20483c2d4ed50eb6bfb0fb9ffbb552a2e090be9754ab9204715e95"}
Nov 03 20:44:05.949300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:05.949206    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podStartSLOduration=1.949191906 podStartE2EDuration="1.949191906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:05.948542471 +0000 UTC m=+870.875112707" watchObservedRunningTime="2026-11-03 20:44:05.949191906 +0000 UTC m=+870.875762173"
Nov 03 20:44:06.299912 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:06.299884    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:15.299772 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:15.299738    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:15.311720 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:15.311697    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:15.920524 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:44:15.920493    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:44:27.394009 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:44:27.393969    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:44:57.365732 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:44:57.365695    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:45:02.751899 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.751866    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"]
Nov 03 20:45:02.755611 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.755593    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.804368 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.804343    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:02.840283 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.840256    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.840410 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.840315    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.840454 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.840432    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.850640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.850617    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 20:45:02.852803 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.852785    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"]
Nov 03 20:45:02.941291 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.941266    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.941432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.941352    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.941432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.941402    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.942023 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.941991    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.943687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.943666    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:02.999384 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:02.999365    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\") pod \"maas-api-revoke-sub-8a3a94dc-kpqqt\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:03.068169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:03.068137    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:03.435539 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:03.435279    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"]
Nov 03 20:45:03.440844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:03.440820    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:45:04.180046 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:04.180001    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt" event={"ID":"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b","Type":"ContainerStarted","Data":"1ac43725c4683e4275cdfaecd31dae594e2034fa976f7a05b236749176b95fde"}
Nov 03 20:45:08.202992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:08.202954    4274 generic.go:356] "Generic (PLEG): container finished" podID="d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" containerID="bea1e1ebcfe98b2fa947f85edc3ccd22c47dc11946d4f6e1c7331dff94707bc7" exitCode=0
Nov 03 20:45:08.203383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:08.202995    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt" event={"ID":"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b","Type":"ContainerDied","Data":"bea1e1ebcfe98b2fa947f85edc3ccd22c47dc11946d4f6e1c7331dff94707bc7"}
Nov 03 20:45:09.338357 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.338334    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:09.409082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.409054    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\") pod \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") "
Nov 03 20:45:09.409206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.409117    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\") pod \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") "
Nov 03 20:45:09.409206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.409167    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\") pod \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\" (UID: \"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b\") "
Nov 03 20:45:09.409447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.409423    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca" pod "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" (UID: "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:09.411141 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.411120    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca" pod "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" (UID: "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:09.411244 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.411167    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95" pod "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" (UID: "d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b"). InnerVolumeSpecName "kube-api-access-pnc95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:09.510148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.510121    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-xks-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:09.510148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.510144    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-maas-api-openshift-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:09.510320 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:09.510154    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnc95\" (UniqueName: \"kubernetes.io/projected/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b-kube-api-access-pnc95\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:10.214095 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:10.214051    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"
Nov 03 20:45:10.214257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:10.214054    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt" event={"ID":"d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b","Type":"ContainerDied","Data":"1ac43725c4683e4275cdfaecd31dae594e2034fa976f7a05b236749176b95fde"}
Nov 03 20:45:10.214257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:10.214178    4274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac43725c4683e4275cdfaecd31dae594e2034fa976f7a05b236749176b95fde"
Nov 03 20:45:15.195063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.194953    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 20:45:15.195646 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.195626    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" containerName="revoke-keys"
Nov 03 20:45:15.195749 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.195647    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" containerName="revoke-keys"
Nov 03 20:45:15.195804 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.195763    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" containerName="revoke-keys"
Nov 03 20:45:15.199558 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.199536    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.267939 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.267907    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.268143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.267987    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.268143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.268015    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.268143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.268074    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqrn\" (UniqueName: \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.268311 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.268158    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.268436 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.268411    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.295258 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.295205    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"default-dockercfg-7wjqd\""
Nov 03 20:45:15.295446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.295260    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"premium-w0-1e6fe1-kserve-self-signed-certs\""
Nov 03 20:45:15.295446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.295354    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"kube-root-ca.crt\""
Nov 03 20:45:15.295446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.295355    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"openshift-service-ca.crt\""
Nov 03 20:45:15.298989 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.298969    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 20:45:15.369515 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369484    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.369713 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369552    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.369713 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369571    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.369713 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369599    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqrn\" (UniqueName: \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.369878 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369736    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.369878 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.369808    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.370061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.370026    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.370143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.370093    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.370143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.370119    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.371880 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.371857    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.372015 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.372001    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.427611 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.427582    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqrn\" (UniqueName: \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\") pod \"premium-w0-1e6fe1-kserve-6b96764b74-p2rxj\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") " pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.511637 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.511608    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:15.879248 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:15.879224    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 20:45:15.880627 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:45:15.880595    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8969180_8963_449a_b5cd_93a6524a864c.slice/crio-b143d54665b3c9fcf82e18624cbb9cc1b6a57d6e2ada9b87877b73785d6e6700 WatchSource:0}: Error finding container b143d54665b3c9fcf82e18624cbb9cc1b6a57d6e2ada9b87877b73785d6e6700: Status 404 returned error can't find the container with id b143d54665b3c9fcf82e18624cbb9cc1b6a57d6e2ada9b87877b73785d6e6700
Nov 03 20:45:16.245587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:16.245508    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" event={"ID":"b8969180-8963-449a-b5cd-93a6524a864c","Type":"ContainerStarted","Data":"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"}
Nov 03 20:45:16.245587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:16.245550    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" event={"ID":"b8969180-8963-449a-b5cd-93a6524a864c","Type":"ContainerStarted","Data":"b143d54665b3c9fcf82e18624cbb9cc1b6a57d6e2ada9b87877b73785d6e6700"}
Nov 03 20:45:16.360981 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:16.360931    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podStartSLOduration=1.360917859 podStartE2EDuration="1.360917859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:16.359344221 +0000 UTC m=+941.285914459" watchObservedRunningTime="2026-11-03 20:45:16.360917859 +0000 UTC m=+941.287488095"
Nov 03 20:45:16.512716 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:16.512685    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:25.512529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:25.512499    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:25.525194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:25.525169    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:26.306577 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:26.306543    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:45:27.370999 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:45:27.370954    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:45:30.672821 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.672783    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 20:45:30.676439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.676418    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.725452 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.725421    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"kube-root-ca.crt\""
Nov 03 20:45:30.771189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.771161    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"openshift-service-ca.crt\""
Nov 03 20:45:30.771378 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.771272    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"default-dockercfg-kzx24\""
Nov 03 20:45:30.771444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.771389    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"premium-w1-0cd49b-kserve-self-signed-certs\""
Nov 03 20:45:30.773786 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.773761    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.823795    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.823877    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.823928    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8hx\" (UniqueName: \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.823967    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.824006    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.825503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.824068    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.924915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.924829    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.924915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.924880    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.924915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.924912    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8hx\" (UniqueName: \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.924938    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.924962    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.925112    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.925304    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925422 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.925366    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.925470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.925414    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.927223 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.927197    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.927385 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.927370    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.982268 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.982234    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8hx\" (UniqueName: \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\") pod \"premium-w1-0cd49b-kserve-5f7bff97b7-w77bq\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") " pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:30.988084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:30.988063    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:31.353623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:31.353594    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 20:45:31.354969 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:45:31.354939    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09b80b7f_fbf8_44b4_9be5_035cb061d4e6.slice/crio-ae223473c0d6530c875034aa061cbac94c84cecb0b7fa787283d623dde8b314a WatchSource:0}: Error finding container ae223473c0d6530c875034aa061cbac94c84cecb0b7fa787283d623dde8b314a: Status 404 returned error can't find the container with id ae223473c0d6530c875034aa061cbac94c84cecb0b7fa787283d623dde8b314a
Nov 03 20:45:32.335758 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:32.335724    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" event={"ID":"09b80b7f-fbf8-44b4-9be5-035cb061d4e6","Type":"ContainerStarted","Data":"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"}
Nov 03 20:45:32.336160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:32.335768    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" event={"ID":"09b80b7f-fbf8-44b4-9be5-035cb061d4e6","Type":"ContainerStarted","Data":"ae223473c0d6530c875034aa061cbac94c84cecb0b7fa787283d623dde8b314a"}
Nov 03 20:45:32.445103 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:32.445027    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podStartSLOduration=2.445013032 podStartE2EDuration="2.445013032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:32.444284186 +0000 UTC m=+957.370854422" watchObservedRunningTime="2026-11-03 20:45:32.445013032 +0000 UTC m=+957.371583269"
Nov 03 20:45:32.988729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:32.988696    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:40.988969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:40.988937    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:41.001284 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:41.001258    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:41.391702 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:45:41.391672    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:45:57.365435 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:45:57.365402    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:46:27.365396 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:46:27.365354    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:46:57.365412 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:46:57.365367    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:47:24.280583 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.280547    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 20:47:24.284373 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.284353    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.333114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.333095    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3b8feb-openshift-default-dockercfg-q9mnd\""
Nov 03 20:47:24.356992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.356961    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357123 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357010    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357123 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357087    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357234 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357208    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357296 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357275    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357351 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357335    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357376    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tmv\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357494 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357477    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.357562 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.357522    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.381133 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.381111    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 20:47:24.458327 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458300    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tmv\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458455 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458334    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458455 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458368    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458554 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458532    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458607 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458571    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458661 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458606    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458715    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458793 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458734    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458793 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458774    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458803    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.458955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.458898    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.459021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.459000    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.459158 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.459136    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.459543 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.459519    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.460881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.460856    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.461030 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.461015    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.515137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.515115    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.515319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.515300    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tmv\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.597260 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.597205    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:24.710235 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.710206    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 20:47:24.714334 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.714310    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762507    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762558    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762647    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762674    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762863 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762791    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762863 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762834    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.762961 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.762869    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.763070 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.763023    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.763139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.763069    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3d2008-openshift-default-dockercfg-z445t\""
Nov 03 20:47:24.763139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.763084    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl88s\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.772738 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.772715    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 20:47:24.863710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863649    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863694    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863720    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863738    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863776    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863807    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.863903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863834    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863911    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.863944    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl88s\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.864147    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.864192    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864351 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.864329    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.864406    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.864550 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.864535    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.866046 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.866014    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.866297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.866278    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.920495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.920463    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:24.920660 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:24.920643    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl88s\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:25.011340 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.011313    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 20:47:25.012440 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:47:25.012412    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aeba99_5b4b_49f0_95e1_bf2d2fd60c8f.slice/crio-be4f4ce7a6996b21889072ef207e582ad35d5f9e34c8e265b4f60c925815a10e WatchSource:0}: Error finding container be4f4ce7a6996b21889072ef207e582ad35d5f9e34c8e265b4f60c925815a10e: Status 404 returned error can't find the container with id be4f4ce7a6996b21889072ef207e582ad35d5f9e34c8e265b4f60c925815a10e
Nov 03 20:47:25.014670 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.014639    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:25.014754 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.014700    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:25.014754 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.014729    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:25.027746 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.027701    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:25.448341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.448315    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 20:47:25.449710 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:47:25.449678    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07127ca8_63d9_4854_807b_3c19ad32238b.slice/crio-2bd3c3c0f46a708fc848598e2f1668602b00a599659b2a2b60c2b0622a26e5dd WatchSource:0}: Error finding container 2bd3c3c0f46a708fc848598e2f1668602b00a599659b2a2b60c2b0622a26e5dd: Status 404 returned error can't find the container with id 2bd3c3c0f46a708fc848598e2f1668602b00a599659b2a2b60c2b0622a26e5dd
Nov 03 20:47:25.452494 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.452460    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:25.452552 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.452528    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:25.452588 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:25.452555    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:26.020603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.020557    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" event={"ID":"07127ca8-63d9-4854-807b-3c19ad32238b","Type":"ContainerStarted","Data":"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"}
Nov 03 20:47:26.020802 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.020668    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" event={"ID":"07127ca8-63d9-4854-807b-3c19ad32238b","Type":"ContainerStarted","Data":"2bd3c3c0f46a708fc848598e2f1668602b00a599659b2a2b60c2b0622a26e5dd"}
Nov 03 20:47:26.025551 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.025497    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" event={"ID":"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f","Type":"ContainerStarted","Data":"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"}
Nov 03 20:47:26.025856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.025822    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" event={"ID":"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f","Type":"ContainerStarted","Data":"be4f4ce7a6996b21889072ef207e582ad35d5f9e34c8e265b4f60c925815a10e"}
Nov 03 20:47:26.028117 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.028086    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:26.029879 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.029851    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:26.029996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.029912    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:26.146059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.145615    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podStartSLOduration=2.145582106 podStartE2EDuration="2.145582106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:26.142344805 +0000 UTC m=+1071.068915054" watchObservedRunningTime="2026-11-03 20:47:26.145582106 +0000 UTC m=+1071.072152345"
Nov 03 20:47:26.259985 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.258530    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podStartSLOduration=2.258510872 podStartE2EDuration="2.258510872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:26.258390427 +0000 UTC m=+1071.184960665" watchObservedRunningTime="2026-11-03 20:47:26.258510872 +0000 UTC m=+1071.185081102"
Nov 03 20:47:26.605058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.604107    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:26.605058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.604271    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:26.605058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:26.604319    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:27.029074 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:27.029023    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:27.029242 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:27.029099    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:27.412739 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:47:27.412630    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:47:27.599804 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:27.599765    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:27.600122 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:27.600089    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:28.028351 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:28.028315    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:28.028716 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:28.028369    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:28.598527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:28.598490    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:28.598862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:28.598814    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:29.029062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:29.029009    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:29.029625 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:29.029595    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:29.603084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:29.600273    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:29.603084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:29.600324    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:30.028417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:30.028369    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:30.028597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:30.028435    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:30.599667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:30.599152    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:30.599667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:30.599215    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:31.029441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:31.029408    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:31.029616 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:31.029467    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:31.597720 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:31.597690    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:31.597895 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:31.597743    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:47:32.028196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:32.028152    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:32.028711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:32.028222    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:47:32.854271 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:32.854242    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:32.854684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:32.854659    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:32.855537 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:32.855516    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 20:47:33.033255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:33.033224    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:33.033646 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:33.033511    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:33.034137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:33.034121    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 20:47:54.208094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.208061    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 20:47:54.211844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.211825    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.307997 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.307962    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"distinct-w2-3b8feb-kserve-self-signed-certs\""
Nov 03 20:47:54.312186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.312159    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 20:47:54.366562 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.366538    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.366717 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.366690    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.366802 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.366786    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.366864 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.366851    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.366914 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.366896    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.367091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.367064    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854kr\" (UniqueName: \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468384 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468298    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468384 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468343    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468384    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854kr\" (UniqueName: \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468412    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468484    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468538    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468754 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468729    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468825    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.468891 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.468842    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.470772 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.470751    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.470946 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.470930    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.525513 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.525485    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854kr\" (UniqueName: \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\") pod \"distinct-w2-3b8feb-kserve-755948487-7vxvk\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:54.824953 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:54.824919    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:55.197593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:55.197567    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 20:47:55.198595 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:47:55.198558    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41ad8f9d_06ff_4fda_8b9f_506634484206.slice/crio-0bdde147f5d1837adc4d836d03a9a65d13306c392545c5992ef4e89734daf6c4 WatchSource:0}: Error finding container 0bdde147f5d1837adc4d836d03a9a65d13306c392545c5992ef4e89734daf6c4: Status 404 returned error can't find the container with id 0bdde147f5d1837adc4d836d03a9a65d13306c392545c5992ef4e89734daf6c4
Nov 03 20:47:55.214903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:55.214876    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" event={"ID":"41ad8f9d-06ff-4fda-8b9f-506634484206","Type":"ContainerStarted","Data":"0bdde147f5d1837adc4d836d03a9a65d13306c392545c5992ef4e89734daf6c4"}
Nov 03 20:47:56.218382 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:56.218344    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" event={"ID":"41ad8f9d-06ff-4fda-8b9f-506634484206","Type":"ContainerStarted","Data":"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"}
Nov 03 20:47:56.327771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:56.327724    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podStartSLOduration=2.327710939 podStartE2EDuration="2.327710939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.327292056 +0000 UTC m=+1101.253862293" watchObservedRunningTime="2026-11-03 20:47:56.327710939 +0000 UTC m=+1101.254281175"
Nov 03 20:47:56.825734 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:47:56.825702    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:47:57.365001 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:47:57.364966    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:48:04.825260 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:04.825221    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:48:04.838321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:04.838295    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:48:05.307281 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:05.307253    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:48:06.940903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:06.940872    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 20:48:06.945904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:06.945882    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:06.994566 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:06.994537    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"openshift-service-ca.crt\""
Nov 03 20:48:07.011761 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.011732    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.011913 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.011796    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.011980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.011940    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.012092 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.012075    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.012162 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.012112    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4vj\" (UniqueName: \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.012264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.012238    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.040603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.040578    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"kube-root-ca.crt\""
Nov 03 20:48:07.040603 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.040609    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"default-dockercfg-9dx65\""
Nov 03 20:48:07.040777 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.040651    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"distinct-w5-2a5ec9-kserve-self-signed-certs\""
Nov 03 20:48:07.043455 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.043435    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 20:48:07.113408 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113374    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113447    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113503    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113575    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113611    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4vj\" (UniqueName: \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113652    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113885 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113820    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.113957 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113935    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.114022 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.113997    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.115663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.115637    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.116244 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.116224    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.172782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.172758    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4vj\" (UniqueName: \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\") pod \"distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.257640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.257618    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:07.647883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:07.647857    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 20:48:07.649761 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:48:07.649729    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeff67ee_3424_465b_935f_284debb0350a.slice/crio-703795a601d151efa9ef4dde4f692d31c4dc51e9fadc8b3d631b0fff8317edbf WatchSource:0}: Error finding container 703795a601d151efa9ef4dde4f692d31c4dc51e9fadc8b3d631b0fff8317edbf: Status 404 returned error can't find the container with id 703795a601d151efa9ef4dde4f692d31c4dc51e9fadc8b3d631b0fff8317edbf
Nov 03 20:48:08.313105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:08.313066    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" event={"ID":"feff67ee-3424-465b-935f-284debb0350a","Type":"ContainerStarted","Data":"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"}
Nov 03 20:48:08.313503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:08.313122    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" event={"ID":"feff67ee-3424-465b-935f-284debb0350a","Type":"ContainerStarted","Data":"703795a601d151efa9ef4dde4f692d31c4dc51e9fadc8b3d631b0fff8317edbf"}
Nov 03 20:48:08.423689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:08.423637    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podStartSLOduration=2.423624631 podStartE2EDuration="2.423624631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:08.421444256 +0000 UTC m=+1113.348014494" watchObservedRunningTime="2026-11-03 20:48:08.423624631 +0000 UTC m=+1113.350194868"
Nov 03 20:48:09.258716 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:09.258686    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:13.491999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:13.491963    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:48:13.492433 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:13.492185    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy" containerID="cri-o://a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3" gracePeriod=30
Nov 03 20:48:16.625680 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.625601    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 20:48:16.630109 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.630087    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.724584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.724540    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"distinct-w4-3d2008-kserve-self-signed-certs\""
Nov 03 20:48:16.727432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727396    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.727597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727575    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.727671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727631    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.727731 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727720    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.727791 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727770    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrn6t\" (UniqueName: \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.727853 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.727837    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.728779 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.728758    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 20:48:16.829396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829348    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829403    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829692 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829462    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829692 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829499    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrn6t\" (UniqueName: \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829692 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829543    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829692 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829584    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829907 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829805    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829886    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.829969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.829956    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.832350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.832322    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.832532 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.832514    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.888456 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.888400    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrn6t\" (UniqueName: \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\") pod \"distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:16.942590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:16.942561    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:17.258144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:17.258109    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:17.270509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:17.270483    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:17.310612 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:17.310583    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 20:48:17.312304 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:48:17.312278    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod630ef936_48ad_44bc_aa99_e35d2f503099.slice/crio-65f3d5f7f8114234e0c2363e874f29a012923fbdcf5177a74716be015cc807a4 WatchSource:0}: Error finding container 65f3d5f7f8114234e0c2363e874f29a012923fbdcf5177a74716be015cc807a4: Status 404 returned error can't find the container with id 65f3d5f7f8114234e0c2363e874f29a012923fbdcf5177a74716be015cc807a4
Nov 03 20:48:17.364340 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:17.364316    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" event={"ID":"630ef936-48ad-44bc-aa99-e35d2f503099","Type":"ContainerStarted","Data":"65f3d5f7f8114234e0c2363e874f29a012923fbdcf5177a74716be015cc807a4"}
Nov 03 20:48:17.375724 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:17.375704    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:48:18.371493 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:18.371447    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" event={"ID":"630ef936-48ad-44bc-aa99-e35d2f503099","Type":"ContainerStarted","Data":"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"}
Nov 03 20:48:18.482679 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:18.482629    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podStartSLOduration=2.482615881 podStartE2EDuration="2.482615881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:18.479917428 +0000 UTC m=+1123.406487664" watchObservedRunningTime="2026-11-03 20:48:18.482615881 +0000 UTC m=+1123.409186120"
Nov 03 20:48:18.943687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:18.943661    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:19.153089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.153069    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:48:19.263116 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263092    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263126    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263155    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263195    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263214    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263262    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263417    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263487    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263413    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:19.263621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263553    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\") pod \"c840bef8-ed6c-473b-99d3-ff601513d093\" (UID: \"c840bef8-ed6c-473b-99d3-ff601513d093\") "
Nov 03 20:48:19.263861 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263465    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:19.263990 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263621    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:19.263990 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263752    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:19.263990 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.263678    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:19.264187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.264060    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.264187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.264083    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.264187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.264099    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.264187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.264113    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c840bef8-ed6c-473b-99d3-ff601513d093-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.264187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.264127    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.265497 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.265471    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:48:19.265608 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.265530    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "kube-api-access-rkj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:19.265897 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.265875    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:19.265987 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.265899    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token" pod "c840bef8-ed6c-473b-99d3-ff601513d093" (UID: "c840bef8-ed6c-473b-99d3-ff601513d093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:19.365006 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.364971    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.365006 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.365000    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c840bef8-ed6c-473b-99d3-ff601513d093-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.365006 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.365010    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c840bef8-ed6c-473b-99d3-ff601513d093-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.365250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.365019    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkj5k\" (UniqueName: \"kubernetes.io/projected/c840bef8-ed6c-473b-99d3-ff601513d093-kube-api-access-rkj5k\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.378871 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.378840    4274 generic.go:356] "Generic (PLEG): container finished" podID="c840bef8-ed6c-473b-99d3-ff601513d093" containerID="a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3" exitCode=0
Nov 03 20:48:19.379236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.378885    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"
Nov 03 20:48:19.379236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.378923    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" event={"ID":"c840bef8-ed6c-473b-99d3-ff601513d093","Type":"ContainerDied","Data":"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"}
Nov 03 20:48:19.379236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.378973    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs" event={"ID":"c840bef8-ed6c-473b-99d3-ff601513d093","Type":"ContainerDied","Data":"0f525f6e3763064e0f2047951393ef4ebb0294553c81a982d0dada7f2068fceb"}
Nov 03 20:48:19.379236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.378997    4274 scope.go:122] "RemoveContainer" containerID="a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"
Nov 03 20:48:19.390137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.390112    4274 scope.go:122] "RemoveContainer" containerID="a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"
Nov 03 20:48:19.390399 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:48:19.390381    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3\": container with ID starting with a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3 not found: ID does not exist" containerID="a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"
Nov 03 20:48:19.390442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.390409    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3"} err="failed to get container status \"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3\": rpc error: code = NotFound desc = could not find container \"a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3\": container with ID starting with a22b997f725f98b52f1bbd32c1fb92a4045007b9a20057eeb6f779e7231613e3 not found: ID does not exist"
Nov 03 20:48:19.532595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.532522    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:48:19.536905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:19.536878    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jpxgs"]
Nov 03 20:48:21.213527 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:21.213484    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" path="/var/lib/kubelet/pods/c840bef8-ed6c-473b-99d3-ff601513d093/volumes"
Nov 03 20:48:26.943210 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:26.943165    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:26.955447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:26.955422    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:27.365570 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:48:27.365535    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:48:27.432465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:27.432440    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:48:30.194537 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.194499    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 20:48:30.195001 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.194987    4274 cpu_manager.go:414] "containerMap: removing container" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy"
Nov 03 20:48:30.195070 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.195002    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy"
Nov 03 20:48:30.195132 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.195122    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c840bef8-ed6c-473b-99d3-ff601513d093" containerName="istio-proxy"
Nov 03 20:48:30.199651 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.199631    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.270793 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.270760    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.270983 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.270820    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.271085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.270978    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.271152 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.271078    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgt4g\" (UniqueName: \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.271152 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.271118    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.271365 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.271348    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.294514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.294487    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"distinct-w3-88228a-kserve-self-signed-certs\""
Nov 03 20:48:30.297446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.297420    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 20:48:30.371961 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.371928    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372070    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372115    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372144    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372190    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372239    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgt4g\" (UniqueName: \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372423    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372517    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.372595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.372541    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.374411 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.374383    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.374822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.374800    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.429551 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.429527    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgt4g\" (UniqueName: \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\") pod \"distinct-w3-88228a-kserve-55b5fbb57-zhdc6\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.511964 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.511939    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:30.881482 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:30.881451    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 20:48:30.883591 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:48:30.883562    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47623b5e_6801_4f45_9c81_6715852ab597.slice/crio-ee4f246f05a97049d4f4a5368f29d15947263aacabe072b4d8640c1da49ea028 WatchSource:0}: Error finding container ee4f246f05a97049d4f4a5368f29d15947263aacabe072b4d8640c1da49ea028: Status 404 returned error can't find the container with id ee4f246f05a97049d4f4a5368f29d15947263aacabe072b4d8640c1da49ea028
Nov 03 20:48:31.445796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:31.445762    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" event={"ID":"47623b5e-6801-4f45-9c81-6715852ab597","Type":"ContainerStarted","Data":"dc525f844ade43b7d4e3448ac38ae27b6b2e8cf0c6d0c7dc192821e3217f5802"}
Nov 03 20:48:31.445796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:31.445806    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" event={"ID":"47623b5e-6801-4f45-9c81-6715852ab597","Type":"ContainerStarted","Data":"ee4f246f05a97049d4f4a5368f29d15947263aacabe072b4d8640c1da49ea028"}
Nov 03 20:48:31.512842 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:31.512812    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:31.556181 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:31.556118    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podStartSLOduration=1.55609861 podStartE2EDuration="1.55609861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:31.555814058 +0000 UTC m=+1136.482384295" watchObservedRunningTime="2026-11-03 20:48:31.55609861 +0000 UTC m=+1136.482668927"
Nov 03 20:48:40.513003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:40.512970    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:40.532255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:40.532227    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:41.518498 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:41.518465    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:48:57.365072 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:48:57.365020    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:48:58.299959 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.299926    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 20:48:58.312297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.312264    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.359652 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.359618    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 20:48:58.402354 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.402325    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"distinct-2-w0-1e6fe1-kserve-self-signed-certs\""
Nov 03 20:48:58.460915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.460880    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.461100 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.460946    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.461100 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.461086    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txwg\" (UniqueName: \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.461266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.461138    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.461266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.461205    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.461347 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.461335    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.562778 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562687    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.562778 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562762    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563011 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562825    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563011 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562863    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563011 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562918    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txwg\" (UniqueName: \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563011 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.562968    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563270 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.563240    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563326 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.563290    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.563380 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.563363    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.565004 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.564984    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.565154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.565138    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.620295 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.620271    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txwg\" (UniqueName: \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\") pod \"distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.624210 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.624185    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:58.990171 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:58.990145    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 20:48:58.991993 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:48:58.991953    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0036e8ea_399b_42ba_96dd_f2034df77690.slice/crio-b0f76171757034b07754fb4917179fc55ad7c808b9ecf9c1c679addab7dc9338 WatchSource:0}: Error finding container b0f76171757034b07754fb4917179fc55ad7c808b9ecf9c1c679addab7dc9338: Status 404 returned error can't find the container with id b0f76171757034b07754fb4917179fc55ad7c808b9ecf9c1c679addab7dc9338
Nov 03 20:48:59.607915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:59.607874    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" event={"ID":"0036e8ea-399b-42ba-96dd-f2034df77690","Type":"ContainerStarted","Data":"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"}
Nov 03 20:48:59.607915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:59.607923    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" event={"ID":"0036e8ea-399b-42ba-96dd-f2034df77690","Type":"ContainerStarted","Data":"b0f76171757034b07754fb4917179fc55ad7c808b9ecf9c1c679addab7dc9338"}
Nov 03 20:48:59.624313 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:59.624290    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:48:59.719425 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:48:59.719374    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podStartSLOduration=1.719359168 podStartE2EDuration="1.719359168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:59.716623345 +0000 UTC m=+1164.643193583" watchObservedRunningTime="2026-11-03 20:48:59.719359168 +0000 UTC m=+1164.645929404"
Nov 03 20:49:08.624432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:08.624396    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:49:08.642138 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:08.642112    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:49:08.665799 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:08.665772    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:49:09.815266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.815236    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 20:49:09.819389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.819369    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884445 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884414    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884445 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884449    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884468    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884615    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884720 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884644    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884758 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884721    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884848 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884828    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.884895 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.884878    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.885027 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.885002    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xnf\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.918209 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.918170    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 20:49:09.986315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986280    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xnf\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986329    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986346    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986460    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986549    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986597    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986672    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986719    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986742    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986917 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986827    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986917 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986846    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.986999 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.986939    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.987122 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.987093    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.987219 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.987142    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.988948 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.988928    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:09.989091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:09.989075    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:10.044108 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.044083    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xnf\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:10.044108 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.044103    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:10.136083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.136005    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:10.555969 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.555930    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 20:49:10.563691 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.563644    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:10.563803 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.563777    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:10.563875 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.563858    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:10.668327 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.668294    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" event={"ID":"1606d4c8-0342-47b2-8258-d5f019552a36","Type":"ContainerStarted","Data":"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"}
Nov 03 20:49:10.668462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.668352    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" event={"ID":"1606d4c8-0342-47b2-8258-d5f019552a36","Type":"ContainerStarted","Data":"af2b873b1a03a4681e204ecc839a0407aca0d402d59d5aba96de368e38248450"}
Nov 03 20:49:10.783096 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:10.783019    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podStartSLOduration=1.7830019529999999 podStartE2EDuration="1.783001953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:10.780526459 +0000 UTC m=+1175.707096697" watchObservedRunningTime="2026-11-03 20:49:10.783001953 +0000 UTC m=+1175.709572192"
Nov 03 20:49:11.136852 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:11.136817    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:11.138528 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:11.138502    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:11.138642 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:11.138554    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:12.136758 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:12.136724    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:12.136909 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:12.136776    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:13.137360 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:13.137330    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:13.137742 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:13.137389    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:14.136639 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:14.136597    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:14.136835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:14.136657    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:15.136675 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:15.136637    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:15.137079 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:15.136701    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:16.137188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:16.137151    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:49:16.137594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:16.137216    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 20:49:17.449306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:17.449276    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:17.449791 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:17.449774    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:17.450766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:17.450745    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 20:49:27.365670 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:49:27.365616    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:49:43.449302 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.449269    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 20:49:43.452813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.452785    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.543645 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.543612    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"distinct-2-w2-3b8feb-kserve-self-signed-certs\""
Nov 03 20:49:43.552107 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.552078    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 20:49:43.587847 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.587822    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.587994 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.587877    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blct\" (UniqueName: \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.587994 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.587950    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.588115 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.588018    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.588115 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.588062    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.588255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.588235    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689405    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689515    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689561    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blct\" (UniqueName: \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689597    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689640 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689622    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689870 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689641    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689871    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689976 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689938    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.689976 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.689960    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.692068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.692028    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.692330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.692308    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.747784 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.747725    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blct\" (UniqueName: \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\") pod \"distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") " pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:43.764105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:43.764080    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:44.130196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:44.130167    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 20:49:44.131597 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:49:44.131567    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248f67e5_66b7_4b53_9e09_86a28e073482.slice/crio-1481016fea75cd2382cdb74a2819087d88f87b99e49ea4edc9c897289b76c7c3 WatchSource:0}: Error finding container 1481016fea75cd2382cdb74a2819087d88f87b99e49ea4edc9c897289b76c7c3: Status 404 returned error can't find the container with id 1481016fea75cd2382cdb74a2819087d88f87b99e49ea4edc9c897289b76c7c3
Nov 03 20:49:44.860988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:44.860955    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" event={"ID":"248f67e5-66b7-4b53-9e09-86a28e073482","Type":"ContainerStarted","Data":"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"}
Nov 03 20:49:44.861419 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:44.860996    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" event={"ID":"248f67e5-66b7-4b53-9e09-86a28e073482","Type":"ContainerStarted","Data":"1481016fea75cd2382cdb74a2819087d88f87b99e49ea4edc9c897289b76c7c3"}
Nov 03 20:49:44.972546 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:44.972463    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podStartSLOduration=1.972450898 podStartE2EDuration="1.972450898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:44.970547726 +0000 UTC m=+1209.897117953" watchObservedRunningTime="2026-11-03 20:49:44.972450898 +0000 UTC m=+1209.899021135"
Nov 03 20:49:45.765026 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:45.765000    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:53.765016 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:53.764983    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:53.777395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:53.777369    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:53.919782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:53.919747    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:49:54.897116 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:54.897082    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 20:49:54.901054 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:54.901015    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.000345 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.000318    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 20:49:55.015870 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.015834    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016013 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.015886    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016013 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.015956    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016013 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016001    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016054    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016165    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbl5\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016193    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016277    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.016393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.016296    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117087    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117244 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117128    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117284 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117241    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117326 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117310    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117387 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117367    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117449 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117425    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbl5\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117496 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117465    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117548 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117493    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117554    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117579    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117582    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117722    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117902 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117882    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.117952 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.117932    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.119782 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.119760    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.119930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.119914    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.175951 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.175880    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.176226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.176202    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbl5\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.216075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.216021    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:55.632139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.632112    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 20:49:55.634399 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:49:55.634354    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed4bbe3_f822_4ee1_8adc_24cce400261e.slice/crio-0d164a7c39a89960e9a75b50def09ca453835fe8d903e7624749a1bf15eee346 WatchSource:0}: Error finding container 0d164a7c39a89960e9a75b50def09ca453835fe8d903e7624749a1bf15eee346: Status 404 returned error can't find the container with id 0d164a7c39a89960e9a75b50def09ca453835fe8d903e7624749a1bf15eee346
Nov 03 20:49:55.636695 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.636663    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:55.636773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.636725    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:55.636773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.636753    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:55.921598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.921501    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" event={"ID":"8ed4bbe3-f822-4ee1-8adc-24cce400261e","Type":"ContainerStarted","Data":"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"}
Nov 03 20:49:55.921598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:55.921566    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" event={"ID":"8ed4bbe3-f822-4ee1-8adc-24cce400261e","Type":"ContainerStarted","Data":"0d164a7c39a89960e9a75b50def09ca453835fe8d903e7624749a1bf15eee346"}
Nov 03 20:49:56.037992 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:56.037939    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podStartSLOduration=2.037925025 podStartE2EDuration="2.037925025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:56.035130267 +0000 UTC m=+1220.961700504" watchObservedRunningTime="2026-11-03 20:49:56.037925025 +0000 UTC m=+1220.964495261"
Nov 03 20:49:56.216558 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:56.216477    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:49:56.217941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:56.217916    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:49:56.218069 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:56.217964    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:49:57.216636 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:57.216602    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:49:57.217020 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:57.216660    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:49:57.394878 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:49:57.394842    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:49:58.216934 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:58.216901    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:49:58.217358 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:58.216951    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:49:59.216598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:59.216544    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:49:59.216798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:49:59.216614    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:50:00.216874 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:00.216834    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:50:00.217420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:00.216884    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:50:01.217289 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:01.217257    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:50:01.217685 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:01.217309    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:50:02.216631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:02.216591    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:50:02.216803 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:02.216656    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:50:03.221395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:03.221362    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:50:03.221850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:03.221809    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:50:03.222597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:03.222569    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 20:50:10.073221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:10.073186    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"]
Nov 03 20:50:10.119131 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:10.119101    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3a94dc-kpqqt"]
Nov 03 20:50:11.213747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:11.213714    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b" path="/var/lib/kubelet/pods/d7f3ea53-5e6e-4fdb-ac26-1d5caf76133b/volumes"
Nov 03 20:50:25.596596 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.596561    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:50:25.606317 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.606289    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684328 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684298    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684466 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684355    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684513 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684457    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684513 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684500    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxjg\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684588 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684574    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684598    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684666 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684649    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684685    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.684747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.684714    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.700030 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.700003    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:50:25.786318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786289    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786334    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786390    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786432    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786506    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786552    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxjg\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786605    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786630    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786665    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786957 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786751    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.786957 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.786793    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.787101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.787019    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.787101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.787022    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.787298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.787273    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.788708 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.788688    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.789197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.789177    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.846381 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.846354    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.846491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.846477    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxjg\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:25.922614 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:25.922537    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:26.337576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:26.337543    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:50:26.370781 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:50:26.370756    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf6780e_68d8_4240_ac2f_79b03f05cf7c.slice/crio-c4255f5128e79d07c5912bd59773a2ed76f11ad17fe737210b3333184ae4deab WatchSource:0}: Error finding container c4255f5128e79d07c5912bd59773a2ed76f11ad17fe737210b3333184ae4deab: Status 404 returned error can't find the container with id c4255f5128e79d07c5912bd59773a2ed76f11ad17fe737210b3333184ae4deab
Nov 03 20:50:26.373227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:26.373190    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:50:26.373353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:26.373268    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:50:26.373353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:26.373315    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:50:27.104825 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.104792    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" event={"ID":"bdf6780e-68d8-4240-ac2f-79b03f05cf7c","Type":"ContainerStarted","Data":"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"}
Nov 03 20:50:27.105238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.104847    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" event={"ID":"bdf6780e-68d8-4240-ac2f-79b03f05cf7c","Type":"ContainerStarted","Data":"c4255f5128e79d07c5912bd59773a2ed76f11ad17fe737210b3333184ae4deab"}
Nov 03 20:50:27.314091 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.314016    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podStartSLOduration=2.314004008 podStartE2EDuration="2.314004008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:27.266499141 +0000 UTC m=+1252.193069379" watchObservedRunningTime="2026-11-03 20:50:27.314004008 +0000 UTC m=+1252.240574245"
Nov 03 20:50:27.391524 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:50:27.391443    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:50:27.923368 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.923336    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:27.924554 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.924531    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:27.924655 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:27.924585    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:28.923255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:28.923217    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:28.923664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:28.923267    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:29.923144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:29.923108    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:29.923338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:29.923157    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:30.923489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:30.923455    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:30.923887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:30.923518    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:31.923549 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:31.923514    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:31.923933 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:31.923575    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:32.922875 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:32.922840    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:50:32.923061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:32.922896    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:50:33.927647 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:33.927615    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:33.928161 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:33.928141    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:33.928940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:50:33.928921    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:50:57.366020 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:50:57.365984    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:51:04.277217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.277167    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:51:04.277922 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.277887    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.301415 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.301354    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.369158 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.369111    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.388537 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.388499    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.429155 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.429113    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.513062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.512947    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:04.585521 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:04.585114    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:25.629136 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.629104    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:51:25.632890 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.632867    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760395    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760460    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760567    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760587    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760617    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760643    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760775    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760804    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlpx\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.760948 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.760924    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.862236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862196    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.862427 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862316    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.862427 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862394    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862456    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862488    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862816    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:51:25.863810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.862535    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.863628    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.863668    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.863694    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlpx\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.863810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.863697    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.864101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.863951    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.864101 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.864005    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.864203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.864158    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.864471 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.864446    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.867225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.866856    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.868246 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.868221    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.920146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.920077    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.920424 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.920400    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlpx\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:25.945495 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:25.945470    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:26.361643 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.361618    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:51:26.362948 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:51:26.362919    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5147d69b_fa33_4ae8_a2ed_b9a16b66386b.slice/crio-4010b14a4ed909e74852b940d826a739d763206645bc6591d9aba7480483bd6c WatchSource:0}: Error finding container 4010b14a4ed909e74852b940d826a739d763206645bc6591d9aba7480483bd6c: Status 404 returned error can't find the container with id 4010b14a4ed909e74852b940d826a739d763206645bc6591d9aba7480483bd6c
Nov 03 20:51:26.365395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.365362    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.365485 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.365427    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.365485 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.365456    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.478817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.478080    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" event={"ID":"5147d69b-fa33-4ae8-a2ed-b9a16b66386b","Type":"ContainerStarted","Data":"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"}
Nov 03 20:51:26.478817 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.478177    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" event={"ID":"5147d69b-fa33-4ae8-a2ed-b9a16b66386b","Type":"ContainerStarted","Data":"4010b14a4ed909e74852b940d826a739d763206645bc6591d9aba7480483bd6c"}
Nov 03 20:51:26.495137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.495108    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:51:26.499175 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.499152    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.558321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.558291    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:51:26.572856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.572814    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9wq\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573020 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.572903    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573020 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.572991    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573184 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573164    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573220    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573253    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573310    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573336    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.573469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.573410    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.604926 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.604876    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podStartSLOduration=1.60486273 podStartE2EDuration="1.60486273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:26.602502886 +0000 UTC m=+1311.529073124" watchObservedRunningTime="2026-11-03 20:51:26.60486273 +0000 UTC m=+1311.531432967"
Nov 03 20:51:26.674135 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674026    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674135 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674116    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674241    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674273    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674304    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674356    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674381    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674450    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674518    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9wq\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.674689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674639    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.675082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674694    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.675082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.674919    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.675257 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.675236    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.675356 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.675325    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.676571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.676549    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.676748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.676730    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.731531 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.731500    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9wq\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.731531 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.731520    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gm462\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.812665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.812630    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:26.963119 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:26.954124    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:27.298788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.298748    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:51:27.300092 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:51:27.300063    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013ba7f6_a51d_430b_93c4_ec241723c414.slice/crio-5a92d3e039a6af4bc74180142905c7d01a0f11b911cffe556e6e62fc4c8e7e6c WatchSource:0}: Error finding container 5a92d3e039a6af4bc74180142905c7d01a0f11b911cffe556e6e62fc4c8e7e6c: Status 404 returned error can't find the container with id 5a92d3e039a6af4bc74180142905c7d01a0f11b911cffe556e6e62fc4c8e7e6c
Nov 03 20:51:27.302931 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.302894    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.303079 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.302973    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.303079 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.303014    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.389926 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:51:27.389888    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:51:27.487066 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.487009    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" event={"ID":"013ba7f6-a51d-430b-93c4-ec241723c414","Type":"ContainerStarted","Data":"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"}
Nov 03 20:51:27.487335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.487312    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" event={"ID":"013ba7f6-a51d-430b-93c4-ec241723c414","Type":"ContainerStarted","Data":"5a92d3e039a6af4bc74180142905c7d01a0f11b911cffe556e6e62fc4c8e7e6c"}
Nov 03 20:51:27.601408 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.601292    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podStartSLOduration=1.601274477 podStartE2EDuration="1.601274477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:27.598711793 +0000 UTC m=+1312.525282030" watchObservedRunningTime="2026-11-03 20:51:27.601274477 +0000 UTC m=+1312.527844715"
Nov 03 20:51:27.812954 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.812926    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:27.971118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.954936    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:27.971118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:27.955010    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:28.815252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:28.815216    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.815715 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:28.815272    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:28.967059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:28.962154    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.967059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:28.962214    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:29.817459 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:29.817414    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.817961 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:29.817486    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:29.953059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:29.949379    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.953059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:29.949447    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:30.815073 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:30.813644    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.815073 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:30.813702    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:30.970060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:30.967364    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.970060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:30.967424    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:31.814169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:31.813896    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.814169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:31.813948    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:31.953059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:31.949399    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.953059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:31.949471    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:32.813325 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:32.813282    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.813725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:32.813351    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:32.951058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:32.949410    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.951058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:32.949475    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:51:33.813879 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:33.813847    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.814374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:33.813905    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 20:51:33.950673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:33.950638    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:33.951031 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:33.951012    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:33.951748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:33.951726    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:51:34.817411 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:34.817384    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:34.817898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:34.817886    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:34.818608 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:34.818587    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:51:35.366232 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:35.366201    4274 scope.go:122] "RemoveContainer" containerID="bea1e1ebcfe98b2fa947f85edc3ccd22c47dc11946d4f6e1c7331dff94707bc7"
Nov 03 20:51:57.366420 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:51:57.366384    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:51:57.745448 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.745358    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 20:51:57.749827 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.749801    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.798410 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.798383    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0cd49b-openshift-default-dockercfg-qtdr4\""
Nov 03 20:51:57.849299 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.849263    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 20:51:57.930334 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930301    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930341    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930427    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930560 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930496    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930601 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930588    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930643 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930612    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930730 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930717    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930764 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930754    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcc6\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:57.930852 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:57.930829    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032431 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032405    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032447    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032485    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032531    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032551    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032767 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032570    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032767 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032656    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032767 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032707    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcc6\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032908 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032780    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.032962 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032905    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.033023 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.032973    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.033172 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.033147    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.033334 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.033317    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.033459 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.033442    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.034816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.034795    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.034926 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.034907    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.089583 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.089555    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.089678 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.089623    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcc6\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.363255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.363180    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:51:58.780120 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.780094    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 20:51:58.781683 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:51:58.781644    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0858a92_77a9_4d39_8836_a730a59bc83d.slice/crio-4beb734f90e2e23d03c1d70eb11f43710e97c94da99a55ab96dadc571b4cfd8b WatchSource:0}: Error finding container 4beb734f90e2e23d03c1d70eb11f43710e97c94da99a55ab96dadc571b4cfd8b: Status 404 returned error can't find the container with id 4beb734f90e2e23d03c1d70eb11f43710e97c94da99a55ab96dadc571b4cfd8b
Nov 03 20:51:58.784159 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.784127    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:58.784230 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.784191    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:58.784267 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:58.784226    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:59.721503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:59.721466    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" event={"ID":"b0858a92-77a9-4d39-8836-a730a59bc83d","Type":"ContainerStarted","Data":"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"}
Nov 03 20:51:59.721709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:59.721516    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" event={"ID":"b0858a92-77a9-4d39-8836-a730a59bc83d","Type":"ContainerStarted","Data":"4beb734f90e2e23d03c1d70eb11f43710e97c94da99a55ab96dadc571b4cfd8b"}
Nov 03 20:51:59.882774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:51:59.882713    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podStartSLOduration=2.88269517 podStartE2EDuration="2.88269517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:59.881440113 +0000 UTC m=+1344.808010346" watchObservedRunningTime="2026-11-03 20:51:59.88269517 +0000 UTC m=+1344.809265408"
Nov 03 20:52:00.368298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:00.368264    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:52:00.370197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:00.370163    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:00.370344 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:00.370215    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:01.370248 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:01.370204    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:01.370796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:01.370267    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:02.369636 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:02.369599    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:02.370188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:02.370153    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:03.370060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:03.369610    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:03.370060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:03.369668    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:04.365711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:04.365671    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:04.366492 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:04.366460    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:05.363651 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:05.363611    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:05.364217 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:05.363670    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:52:06.368319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:06.368283    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:52:06.368729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:06.368676    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:52:06.369285 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:06.369262    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 20:52:11.514882 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.514848    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:52:11.519747 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.519722    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.618735 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.618697    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:52:11.620310 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620280    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620335    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngsk\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620447 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620399    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620476    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620513    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620570    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620638    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620662    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.620766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.620723    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.739829 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.739788    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.740205 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.740179    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.740371 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.740334    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.740513 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.740499    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.740710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.740691    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.740844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.740827    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngsk\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.741054 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.741021    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.741240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.741221    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.741377 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.741360    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.741538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.741441    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.741733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.741710    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.755310 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.742899    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.755310 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.744821    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.755310 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.747311    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.755310 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.755255    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.758726 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.758706    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.800009 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.799922    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngsk\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.800474 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.800449    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:11.842684 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:11.842655    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:12.281106 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:52:12.281052    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e0cdfe_2b8c_478d_8f57_1b22a95dd75c.slice/crio-f611c271200de9a67addbe93b78dabf1760848abf38f9bcab5ff6c9f15edfb4c WatchSource:0}: Error finding container f611c271200de9a67addbe93b78dabf1760848abf38f9bcab5ff6c9f15edfb4c: Status 404 returned error can't find the container with id f611c271200de9a67addbe93b78dabf1760848abf38f9bcab5ff6c9f15edfb4c
Nov 03 20:52:12.284810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.284147    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:52:12.287099 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.285892    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.287099 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.285976    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.287099 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.286020    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.902342 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.902245    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" event={"ID":"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c","Type":"ContainerStarted","Data":"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"}
Nov 03 20:52:12.904653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:12.904616    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" event={"ID":"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c","Type":"ContainerStarted","Data":"f611c271200de9a67addbe93b78dabf1760848abf38f9bcab5ff6c9f15edfb4c"}
Nov 03 20:52:13.030351 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:13.030287    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podStartSLOduration=2.030265773 podStartE2EDuration="2.030265773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:13.027609712 +0000 UTC m=+1357.954179952" watchObservedRunningTime="2026-11-03 20:52:13.030265773 +0000 UTC m=+1357.956836015"
Nov 03 20:52:13.845060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:13.844477    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:13.847107 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:13.847074    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:13.847386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:13.847338    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:14.843556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:14.843517    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:14.843957 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:14.843581    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:15.843742 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:15.843703    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:15.844166 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:15.843763    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:16.843411 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:16.843372    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:16.843619 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:16.843445    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:17.843099 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:17.843061    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:17.843469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:17.843126    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:18.843297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:18.843261    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:18.843687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:18.843309    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:19.843631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:19.843593    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:19.844007 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:19.843656    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 20:52:20.847150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:20.847119    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:20.847543 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:20.847387    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:20.848089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:20.848067    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:52:27.366522 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:27.366484    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:52:46.038089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.037973    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6936-8b312c"]
Nov 03 20:52:46.044904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.044877    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:46.070965 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.070939    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wh4\" (UniqueName: \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\") pod \"test-curl-6936-8b312c\" (UID: \"91b56fdf-e011-42ac-831d-5a0ae27b13a8\") " pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:46.094200 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.094170    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2jkh\""
Nov 03 20:52:46.096816 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.096792    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6936-8b312c"]
Nov 03 20:52:46.176239 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.176195    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wh4\" (UniqueName: \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\") pod \"test-curl-6936-8b312c\" (UID: \"91b56fdf-e011-42ac-831d-5a0ae27b13a8\") " pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:46.242225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.242188    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wh4\" (UniqueName: \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\") pod \"test-curl-6936-8b312c\" (UID: \"91b56fdf-e011-42ac-831d-5a0ae27b13a8\") " pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:46.367392 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:46.367304    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:47.143400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.132066    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6936-8b312c"]
Nov 03 20:52:47.230589 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.214828    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:52:47.522993 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.522950    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.599480 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.597181    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.722176 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.722128    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.951114 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.950113    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.979058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:47.978114    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.017586 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.006653    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.250116 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.245185    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6936-8b312c" event={"ID":"91b56fdf-e011-42ac-831d-5a0ae27b13a8","Type":"ContainerStarted","Data":"c3c6ed3aeb6f4ab3ff674550940d6ac38db11b5dffbf3af16d4046f8144551ed"}
Nov 03 20:52:48.262374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.262323    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.326165 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.326122    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.327201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.326423    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.368265 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.368205    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.412553 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.412506    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.437165 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.437115    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.510272 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.510229    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:48.510597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.510553    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.539056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.531874    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.539056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.532613    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:52:48.649600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.609472    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.649600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.609892    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:52:48.649600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.631434    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.755059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.753194    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.755059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.754650    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.879664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.771295    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.879664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.771645    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:52:48.989518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.989476    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.995211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:48.995183    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:52:49.017414 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.016204    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.017414 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.016606    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:52:49.375277 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.375220    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.478186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.478138    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.478967 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.478930    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.483055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.479406    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.508778 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.487694    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.536213 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.536180    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:49.536463 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.536431    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.536831 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.536802    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.612154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.612112    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.649198 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.649103    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.750673 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.716147    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.777060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.762120    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.777060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.764391    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.788059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.777856    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.993133 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:49.989812    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.041464 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.041424    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.118315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.118169    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:52:50.130273 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.130248    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:52:50.130501 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.130484    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:52:50.130690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.130674    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:52:50.130831 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.130818    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:52:50.130964 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.130952    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:52:50.131186 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131171    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:52:50.131339 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131325    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:52:50.131448 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131437    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:52:50.131565 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131555    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 20:52:50.131680 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131668    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 20:52:50.131790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131779    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:52:50.131904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.131893    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 20:52:50.132020 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.132010    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:52:50.147201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.147179    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 20:52:50.147432 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.147406    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:52:50.147623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.147607    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:52:50.148658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.148638    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:52:50.153311 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.153293    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 20:52:50.161514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.161496    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 20:52:50.176283 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:50.176260    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:52:53.129229 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:53.129187    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6936-8b312c" event={"ID":"91b56fdf-e011-42ac-831d-5a0ae27b13a8","Type":"ContainerStarted","Data":"1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a"}
Nov 03 20:52:53.236598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:53.236551    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6936-8b312c" podStartSLOduration=2.157003061 podStartE2EDuration="7.23653562s" totalImagesPullingTime="5.079532559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:53.235764641 +0000 UTC m=+1398.162334877" watchObservedRunningTime="2026-11-03 20:52:53.23653562 +0000 UTC m=+1398.163105858"
Nov 03 20:52:54.688066 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:54.688019    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6936-8b312c"]
Nov 03 20:52:54.695072 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:54.695030    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6936-8b312c"]
Nov 03 20:52:55.145467 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:55.145421    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6936-8b312c" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" containerName="test-curl-6936-8b312c" containerID="cri-o://1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a" gracePeriod=2
Nov 03 20:52:55.193331 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:55.193290    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6936-8b312c\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:55.256741 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:55.256709    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6936-8b312c\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:57.367102 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:57.367065    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:52:57.407130 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.407076    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:57.526317 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.526282    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\" (UniqueName: \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\") pod \"91b56fdf-e011-42ac-831d-5a0ae27b13a8\" (UID: \"91b56fdf-e011-42ac-831d-5a0ae27b13a8\") "
Nov 03 20:52:57.528813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.528781    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4" pod "91b56fdf-e011-42ac-831d-5a0ae27b13a8" (UID: "91b56fdf-e011-42ac-831d-5a0ae27b13a8"). InnerVolumeSpecName "kube-api-access-44wh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:57.628030 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.627994    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44wh4\" (UniqueName: \"kubernetes.io/projected/91b56fdf-e011-42ac-831d-5a0ae27b13a8-kube-api-access-44wh4\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.707806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.707730    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:52:57.708279 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.708265    4274 cpu_manager.go:414] "containerMap: removing container" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" containerName="test-curl-6936-8b312c"
Nov 03 20:52:57.708334 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.708280    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" containerName="test-curl-6936-8b312c"
Nov 03 20:52:57.708395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.708386    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" containerName="test-curl-6936-8b312c"
Nov 03 20:52:57.711728 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.711709    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.806480 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.806449    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:52:57.830326 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830290    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830480 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830334    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830480 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830419    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830559 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830525    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830614 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830599    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7btc\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830659 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830648    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830750 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830736    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830799 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830761    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.830799 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.830782    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932197    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932227 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932228    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932244    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932319    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932342    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932380    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932448    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932494    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7btc\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932556    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932727 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932707    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932884 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932747    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.932884 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.932824    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.933055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.933002    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.933196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.933177    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.934784 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.934756    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.934905 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.934888    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.989506 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.989437    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:57.989629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:57.989547    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7btc\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:58.024528 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.024506    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:52:58.073201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.073078    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 20:52:58.077891 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.077869    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.098501 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.097759    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 20:52:58.103599 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.103234    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.140844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.140816    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 20:52:58.174571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.174416    4274 generic.go:356] "Generic (PLEG): container finished" podID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" containerID="1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a" exitCode=137
Nov 03 20:52:58.174743 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.174597    4274 scope.go:122] "RemoveContainer" containerID="1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a"
Nov 03 20:52:58.174743 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.174712    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:58.182085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.182061    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 20:52:58.197980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.197946    4274 scope.go:122] "RemoveContainer" containerID="1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a"
Nov 03 20:52:58.198287 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:58.198267    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a\": container with ID starting with 1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a not found: ID does not exist" containerID="1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a"
Nov 03 20:52:58.198359 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.198298    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a"} err="failed to get container status \"1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a\": rpc error: code = NotFound desc = could not find container \"1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a\": container with ID starting with 1c9cb324b4536306fb2e2ebe876538bebb6310a50041d6b85d8701a0582d542a not found: ID does not exist"
Nov 03 20:52:58.237470 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:58.237437    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6936-8b312c\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:58.241145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241085    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241132    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241210    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241248    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqrn\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241271    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241440 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241348    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241440 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241405    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241441    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241459    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h588b\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241632 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241579    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241631    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241666    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241815 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241793    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241822    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241845    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.241912 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241886    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241948 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241926    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.241978 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.241958    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.342972 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.342941    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.342972 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.342976    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.342999    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343018    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343067    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343089    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343201    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343251    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343280    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343307    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqrn\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343338    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343380    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343387    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343450    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343473    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.343491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343482    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343563    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343590    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h588b\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343649    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343704    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343741    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343746    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.343777    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.344805 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.344482    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.344805 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.344484    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.344805 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.344690    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.344973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.344953    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.345028 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.344970    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.345650 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.345628    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.346014 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.345991    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.346118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.345995    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.346851 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.346831    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.400207 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.400177    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.400564 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.400493    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqrn\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.400958 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.400937    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.401134 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.401114    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h588b\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.423116 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.423094    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:58.448094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.448072    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:52:58.449177 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:52:58.449150    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6644bca6_51e3_426d_9e37_8dea63ba02aa.slice/crio-8558ccea503f8b261af80ba383f410d5f33895f31431f101268527724479bdae WatchSource:0}: Error finding container 8558ccea503f8b261af80ba383f410d5f33895f31431f101268527724479bdae: Status 404 returned error can't find the container with id 8558ccea503f8b261af80ba383f410d5f33895f31431f101268527724479bdae
Nov 03 20:52:58.452587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.452552    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.452712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.452615    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.452712 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.452644    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.698685 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.698649    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:52:58.801456 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.801379    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 20:52:58.806003 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:52:58.805977    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3035b852_0efc_4043_b0f2_ecd926963754.slice/crio-360ffc7e6fbbe0210222276afdcb5005ed94c72c98de8bd8fea33ce9d73aa8f3 WatchSource:0}: Error finding container 360ffc7e6fbbe0210222276afdcb5005ed94c72c98de8bd8fea33ce9d73aa8f3: Status 404 returned error can't find the container with id 360ffc7e6fbbe0210222276afdcb5005ed94c72c98de8bd8fea33ce9d73aa8f3
Nov 03 20:52:58.809503 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.809461    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.809596 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.809552    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.809664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:58.809602    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:59.091435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.091398    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 20:52:59.101735 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.101700    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:59.101833 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.101766    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:59.101833 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.101800    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:59.192083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.189498    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" event={"ID":"a1622d3b-4d55-415a-827f-8143ce51b5e5","Type":"ContainerStarted","Data":"c51bf1d3a93c817a8abdbd226ead9e613b2a35787fa13041a33099447ea248a1"}
Nov 03 20:52:59.192664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.192538    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" event={"ID":"6644bca6-51e3-426d-9e37-8dea63ba02aa","Type":"ContainerStarted","Data":"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"}
Nov 03 20:52:59.192664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.192595    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" event={"ID":"6644bca6-51e3-426d-9e37-8dea63ba02aa","Type":"ContainerStarted","Data":"8558ccea503f8b261af80ba383f410d5f33895f31431f101268527724479bdae"}
Nov 03 20:52:59.194705 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.194677    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" event={"ID":"3035b852-0efc-4043-b0f2-ecd926963754","Type":"ContainerStarted","Data":"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"}
Nov 03 20:52:59.195391 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.195370    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" event={"ID":"3035b852-0efc-4043-b0f2-ecd926963754","Type":"ContainerStarted","Data":"360ffc7e6fbbe0210222276afdcb5005ed94c72c98de8bd8fea33ce9d73aa8f3"}
Nov 03 20:52:59.219697 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.219657    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" path="/var/lib/kubelet/pods/91b56fdf-e011-42ac-831d-5a0ae27b13a8/volumes"
Nov 03 20:52:59.311304 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.310683    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podStartSLOduration=2.310667426 podStartE2EDuration="2.310667426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:59.304503773 +0000 UTC m=+1404.231074006" watchObservedRunningTime="2026-11-03 20:52:59.310667426 +0000 UTC m=+1404.237237662"
Nov 03 20:52:59.364398 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:52:59.364297    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6936-8b312c\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="91b56fdf-e011-42ac-831d-5a0ae27b13a8" pod="opendatahub/test-curl-6936-8b312c"
Nov 03 20:52:59.424251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.424220    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:52:59.427718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.427682    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:52:59.427965 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:52:59.427936    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:00.026967 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.026930    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:53:00.027729 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.027702    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:00.027809 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.027762    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:00.237219 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.237176    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" event={"ID":"a1622d3b-4d55-415a-827f-8143ce51b5e5","Type":"ContainerStarted","Data":"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"}
Nov 03 20:53:00.385118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.384078    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podStartSLOduration=2.384056862 podStartE2EDuration="2.384056862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:00.369806497 +0000 UTC m=+1405.296376731" watchObservedRunningTime="2026-11-03 20:53:00.384056862 +0000 UTC m=+1405.310627100"
Nov 03 20:53:00.390061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.387058    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podStartSLOduration=2.3870274 podStartE2EDuration="2.3870274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:59.506856849 +0000 UTC m=+1404.433427089" watchObservedRunningTime="2026-11-03 20:53:00.3870274 +0000 UTC m=+1405.313597637"
Nov 03 20:53:00.425124 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.424532    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:00.425124 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.424596    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:00.720126 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.712184    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:53:00.720126 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.712314    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:00.720126 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:00.712357    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:01.028062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.026441    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:01.028062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.026512    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:01.423582 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.423485    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:01.423582 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.423555    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:01.702563 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.702468    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:01.703153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:01.703120    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:02.028095 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.027469    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:02.028095 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.027544    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:02.594154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.590840    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:02.594154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.590902    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:02.719068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.718156    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:02.719068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:02.718230    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:03.021166 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.021094    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6942-1064af"]
Nov 03 20:53:03.025532 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.025502    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:03.025670 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.025556    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:03.029489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.029468    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:03.099665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.099628    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6942-1064af"]
Nov 03 20:53:03.122306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.122272    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2jkh\""
Nov 03 20:53:03.141089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.141009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fqj\" (UniqueName: \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\") pod \"test-curl-6942-1064af\" (UID: \"fa239472-fff1-439a-9a61-f2019e447d24\") " pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:03.242542 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.242510    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fqj\" (UniqueName: \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\") pod \"test-curl-6942-1064af\" (UID: \"fa239472-fff1-439a-9a61-f2019e447d24\") " pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:03.340269 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.339847    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fqj\" (UniqueName: \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\") pod \"test-curl-6942-1064af\" (UID: \"fa239472-fff1-439a-9a61-f2019e447d24\") " pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:03.353072 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.343984    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:03.425089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.424651    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:03.425089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.424707    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:03.736771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.734671    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:03.736771 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:03.734730    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:04.033561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.033525    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:04.033912 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.033879    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:04.427385 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.427346    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:04.427566 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.427421    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:04.641774 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.635894    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6942-1064af"]
Nov 03 20:53:04.669423 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.663159    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:04.704488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.702147    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:04.704488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.702133    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:04.704488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.702207    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:04.704488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.702431    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:04.766073 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.764121    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:04.834069 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:04.831099    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:05.025735 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.025646    4274 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:05.025735 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.025690    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:53:05.418228 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.417223    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6942-1064af" event={"ID":"fa239472-fff1-439a-9a61-f2019e447d24","Type":"ContainerStarted","Data":"c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352"}
Nov 03 20:53:05.418228 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.417275    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6942-1064af" event={"ID":"fa239472-fff1-439a-9a61-f2019e447d24","Type":"ContainerStarted","Data":"39f38877d6b05124e1acee0bd4fafb889e450071d1af950a2c5c3ef863575797"}
Nov 03 20:53:05.424280 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.424255    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:05.424429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.424298    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:53:05.699188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.699091    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:05.699188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:05.699149    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:06.490012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.489979    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:53:06.490854 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.490823    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:53:06.494300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.493781    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:53:06.651022 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.650955    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6942-1064af" podStartSLOduration=4.65093613 podStartE2EDuration="4.65093613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.531974087 +0000 UTC m=+1410.458544329" watchObservedRunningTime="2026-11-03 20:53:06.65093613 +0000 UTC m=+1411.577506370"
Nov 03 20:53:06.699157 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.699125    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:06.699360 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:06.699182    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 20:53:07.015689 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.015654    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:53:07.016125 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.016093    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:53:07.016931 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.016909    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 20:53:07.705174 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.705148    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:53:07.706081 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.705623    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:53:07.706593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:07.706564    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 20:53:09.095894 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.095393    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:53:09.103095 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.103069    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.151822 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.151792    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15564eb1\""
Nov 03 20:53:09.152090 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.151912    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15564eb1-dockercfg-tkhqt\""
Nov 03 20:53:09.155509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.155481    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:53:09.243058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.241712    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:53:09.249089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.245970    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.249089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.246064    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcfj\" (UniqueName: \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.249089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.247360    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.295888 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.295832    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:53:09.348975 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.348668    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.349182 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.348976    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.349182 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.349021    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48j2m\" (UniqueName: \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.349903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.349524    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcfj\" (UniqueName: \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.349903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.349858    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.408883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.408844    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcfj\" (UniqueName: \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\") pod \"payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.416703 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.416669    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:09.451587 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.451552    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.451742 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.451600    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48j2m\" (UniqueName: \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.452269 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.452248    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.509448 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.509413    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48j2m\" (UniqueName: \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\") pod \"payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") " pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.567361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.567334    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:09.794573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.794508    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:53:09.796475 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:09.796450    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7320a3ea_f736_41c0_bfbc_e05413eb5eb9.slice/crio-85fd0638aaf6c7979d74b09e5cfb922ce13c3d15a65dde6c4746edc0e226fd45 WatchSource:0}: Error finding container 85fd0638aaf6c7979d74b09e5cfb922ce13c3d15a65dde6c4746edc0e226fd45: Status 404 returned error can't find the container with id 85fd0638aaf6c7979d74b09e5cfb922ce13c3d15a65dde6c4746edc0e226fd45
Nov 03 20:53:09.940204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:09.940170    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:53:09.941494 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:09.941468    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34954f14_83bc_4774_b11e_17042454fa06.slice/crio-2ce3659e549358b521bde3fbb9836d7189497bbba89100adebb35b55b2a2f374 WatchSource:0}: Error finding container 2ce3659e549358b521bde3fbb9836d7189497bbba89100adebb35b55b2a2f374: Status 404 returned error can't find the container with id 2ce3659e549358b521bde3fbb9836d7189497bbba89100adebb35b55b2a2f374
Nov 03 20:53:10.457731 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.457697    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" event={"ID":"34954f14-83bc-4774-b11e-17042454fa06","Type":"ContainerStarted","Data":"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"}
Nov 03 20:53:10.458202 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.457771    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:10.458202 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.457798    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" event={"ID":"34954f14-83bc-4774-b11e-17042454fa06","Type":"ContainerStarted","Data":"2ce3659e549358b521bde3fbb9836d7189497bbba89100adebb35b55b2a2f374"}
Nov 03 20:53:10.459315 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.459291    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" event={"ID":"7320a3ea-f736-41c0-bfbc-e05413eb5eb9","Type":"ContainerStarted","Data":"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"}
Nov 03 20:53:10.459417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.459338    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:10.459417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.459357    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" event={"ID":"7320a3ea-f736-41c0-bfbc-e05413eb5eb9","Type":"ContainerStarted","Data":"85fd0638aaf6c7979d74b09e5cfb922ce13c3d15a65dde6c4746edc0e226fd45"}
Nov 03 20:53:10.568552 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.568502    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" podStartSLOduration=1.568488195 podStartE2EDuration="1.568488195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:10.565757938 +0000 UTC m=+1415.492328176" watchObservedRunningTime="2026-11-03 20:53:10.568488195 +0000 UTC m=+1415.495058478"
Nov 03 20:53:10.681911 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:10.681857    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" podStartSLOduration=1.681843793 podStartE2EDuration="1.681843793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:10.676971829 +0000 UTC m=+1415.603542066" watchObservedRunningTime="2026-11-03 20:53:10.681843793 +0000 UTC m=+1415.608414029"
Nov 03 20:53:12.512654 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.512614    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 20:53:12.517667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.517645    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.616705 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.616670    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 20:53:12.691417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691385    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691440    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691491    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691568 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691547    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691672 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691621    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwlz\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691826 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691806    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691838    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691855    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.691948 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.691938    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.792806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792739    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.792934 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792806    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.792934 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792858    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.792934 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792883    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.792934 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792929    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793194 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.792964    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwlz\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793204    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793329 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793305    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793335    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793545    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793672 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793238    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793672 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793622    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793691    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.793796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.793710    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.795387 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.795366    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.795613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.795593    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.849702 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.849671    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:12.849820 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:12.849804    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwlz\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:13.137178 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:13.137095    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:13.565190 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:13.565167    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 20:53:13.566855 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:13.566824    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e49cad_4cd6_4d16_8c2a_8b690c5baa46.slice/crio-df7d2ab307fe49158a238a003a1c196ae315653a0413b005e87b336c3a518edb WatchSource:0}: Error finding container df7d2ab307fe49158a238a003a1c196ae315653a0413b005e87b336c3a518edb: Status 404 returned error can't find the container with id df7d2ab307fe49158a238a003a1c196ae315653a0413b005e87b336c3a518edb
Nov 03 20:53:13.569561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:13.569529    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.569628 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:13.569596    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.569628 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:13.569622    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:14.010700 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.010667    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:53:14.016659 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.016634    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.070975 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.070942    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:53:14.111672 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.111633    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f9337cba-dockercfg-67965\""
Nov 03 20:53:14.111860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.111703    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f9337cba\""
Nov 03 20:53:14.118076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.118046    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l487t\" (UniqueName: \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.118256 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.118228    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.147166 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.147139    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:53:14.151372 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.151352    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.209056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.208344    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:53:14.219504 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.219471    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l487t\" (UniqueName: \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.219738 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.219598    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.220761 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.220729    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.279662 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.279394    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l487t\" (UniqueName: \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\") pod \"payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.320680 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.320650    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.320856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.320731    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pn5\" (UniqueName: \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.333383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.333358    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:14.422458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.422414    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.422688 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.422542    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pn5\" (UniqueName: \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.446565 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.443460    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.489633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.489455    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pn5\" (UniqueName: \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\") pod \"payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") " pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:14.506074 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.502704    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" event={"ID":"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46","Type":"ContainerStarted","Data":"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"}
Nov 03 20:53:14.506074 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.502775    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" event={"ID":"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46","Type":"ContainerStarted","Data":"df7d2ab307fe49158a238a003a1c196ae315653a0413b005e87b336c3a518edb"}
Nov 03 20:53:14.618469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.618359    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podStartSLOduration=2.618346162 podStartE2EDuration="2.618346162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:14.615221186 +0000 UTC m=+1419.541791424" watchObservedRunningTime="2026-11-03 20:53:14.618346162 +0000 UTC m=+1419.544916400"
Nov 03 20:53:14.735188 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.735162    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:53:14.735966 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:14.735939    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9d5973_2866_4183_92c1_0b3b2e930ab1.slice/crio-2e38b45e42442055ef405edbf8289574a9a7fe678ce3d00e1ec3d2f6841358d5 WatchSource:0}: Error finding container 2e38b45e42442055ef405edbf8289574a9a7fe678ce3d00e1ec3d2f6841358d5: Status 404 returned error can't find the container with id 2e38b45e42442055ef405edbf8289574a9a7fe678ce3d00e1ec3d2f6841358d5
Nov 03 20:53:14.764728 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:14.764703    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:15.137297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.137261    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:15.138661 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.138635    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:15.138798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.138685    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:15.174296 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.174160    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:53:15.176024 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:15.175989    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6355981_58fa_4b5a_ad69_82a5a595de73.slice/crio-910bb2cafdebb0da05b5d9a41c8f6ad64179e7363a346086777cb911f48e45dc WatchSource:0}: Error finding container 910bb2cafdebb0da05b5d9a41c8f6ad64179e7363a346086777cb911f48e45dc: Status 404 returned error can't find the container with id 910bb2cafdebb0da05b5d9a41c8f6ad64179e7363a346086777cb911f48e45dc
Nov 03 20:53:15.510827 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.510788    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" event={"ID":"1c9d5973-2866-4183-92c1-0b3b2e930ab1","Type":"ContainerStarted","Data":"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"}
Nov 03 20:53:15.511014 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.510854    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" event={"ID":"1c9d5973-2866-4183-92c1-0b3b2e930ab1","Type":"ContainerStarted","Data":"2e38b45e42442055ef405edbf8289574a9a7fe678ce3d00e1ec3d2f6841358d5"}
Nov 03 20:53:15.511014 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.510884    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:15.512690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.512655    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" event={"ID":"e6355981-58fa-4b5a-ad69-82a5a595de73","Type":"ContainerStarted","Data":"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"}
Nov 03 20:53:15.512820 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.512715    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" event={"ID":"e6355981-58fa-4b5a-ad69-82a5a595de73","Type":"ContainerStarted","Data":"910bb2cafdebb0da05b5d9a41c8f6ad64179e7363a346086777cb911f48e45dc"}
Nov 03 20:53:15.512914 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.512892    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:15.625785 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.625729    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" podStartSLOduration=2.625715842 podStartE2EDuration="2.625715842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:15.623773409 +0000 UTC m=+1420.550343648" watchObservedRunningTime="2026-11-03 20:53:15.625715842 +0000 UTC m=+1420.552286078"
Nov 03 20:53:15.733964 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:15.733914    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" podStartSLOduration=1.733900791 podStartE2EDuration="1.733900791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:15.731889064 +0000 UTC m=+1420.658459301" watchObservedRunningTime="2026-11-03 20:53:15.733900791 +0000 UTC m=+1420.660471027"
Nov 03 20:53:16.137988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:16.137957    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:16.138236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:16.138007    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:17.079083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:17.077230    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6942-1064af"]
Nov 03 20:53:17.079083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:17.077459    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6942-1064af" podUID="fa239472-fff1-439a-9a61-f2019e447d24" containerName="test-curl-6942-1064af" containerID="cri-o://c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352" gracePeriod=2
Nov 03 20:53:17.082097 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:17.082071    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6942-1064af"]
Nov 03 20:53:17.126713 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:53:17.126670    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6942-1064af\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="fa239472-fff1-439a-9a61-f2019e447d24" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:17.138445 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:17.138422    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:17.138551 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:17.138464    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:18.137807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:18.137775    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:18.138202 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:18.137826    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:19.140053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.137833    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:19.140053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.137909    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:19.406330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.406297    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:19.551115 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.551074    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\" (UniqueName: \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\") pod \"fa239472-fff1-439a-9a61-f2019e447d24\" (UID: \"fa239472-fff1-439a-9a61-f2019e447d24\") "
Nov 03 20:53:19.581252 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.579167    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj" pod "fa239472-fff1-439a-9a61-f2019e447d24" (UID: "fa239472-fff1-439a-9a61-f2019e447d24"). InnerVolumeSpecName "kube-api-access-x9fqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:19.586050 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.586003    4274 generic.go:356] "Generic (PLEG): container finished" podID="fa239472-fff1-439a-9a61-f2019e447d24" containerID="c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352" exitCode=137
Nov 03 20:53:19.586338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.586197    4274 scope.go:122] "RemoveContainer" containerID="c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352"
Nov 03 20:53:19.586338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.586162    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:19.618333 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.618157    4274 scope.go:122] "RemoveContainer" containerID="c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352"
Nov 03 20:53:19.625086 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:53:19.618507    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352\": container with ID starting with c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352 not found: ID does not exist" containerID="c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352"
Nov 03 20:53:19.625086 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.618546    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352"} err="failed to get container status \"c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352\": rpc error: code = NotFound desc = could not find container \"c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352\": container with ID starting with c7eff33f414c85a49f34963d2c53a551d9ed1a5da11f779622b453e59c114352 not found: ID does not exist"
Nov 03 20:53:19.659058 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:53:19.658275    4274 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6942-1064af\" is forbidden: User \"system:node:ip-10-0-128-118.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-118.us-west-2.compute.internal' and this object" podUID="fa239472-fff1-439a-9a61-f2019e447d24" pod="opendatahub/test-curl-6942-1064af"
Nov 03 20:53:19.677059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:19.673162    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fqj\" (UniqueName: \"kubernetes.io/projected/fa239472-fff1-439a-9a61-f2019e447d24-kube-api-access-x9fqj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:20.137410 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:20.137375    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:20.137549 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:20.137436    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:21.137648 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.137608    4274 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:53:21.138132 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.137671    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:53:21.215395 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.215356    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa239472-fff1-439a-9a61-f2019e447d24" path="/var/lib/kubelet/pods/fa239472-fff1-439a-9a61-f2019e447d24/volumes"
Nov 03 20:53:21.231656 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.231375    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:53:21.237335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.232272    4274 cpu_manager.go:414] "containerMap: removing container" podUID="fa239472-fff1-439a-9a61-f2019e447d24" containerName="test-curl-6942-1064af"
Nov 03 20:53:21.237335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.232296    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa239472-fff1-439a-9a61-f2019e447d24" containerName="test-curl-6942-1064af"
Nov 03 20:53:21.237335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.232490    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa239472-fff1-439a-9a61-f2019e447d24" containerName="test-curl-6942-1064af"
Nov 03 20:53:21.238001 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.237976    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.292482    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d9a2c67d-openshift-default-dockercfg-vt9vs\""
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300739    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300782    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300814    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300879    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q68x\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300959    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.300994    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.301026    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.301087    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.309024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.301204    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.368430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.368398    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:53:21.403208 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403102    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403208 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403166    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403210    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403253    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403391    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403600 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403567    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403657 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403607    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403657 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403636    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.403762 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.403720    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q68x\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.404264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.404239    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.404667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.404646    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.405417 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.405397    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.408748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.405840    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.408748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.406541    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.408748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.408432    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.408748 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.408551    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.464084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.462839    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.464084 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.463412    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q68x\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\") pod \"e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") " pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:21.468945 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.468917    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:53:21.470641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.470620    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:53:21.583576 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:21.583538    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:22.105708 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.105677    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:53:22.109523 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:22.109498    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93136739_ecc0_4939_a91b_fab66a741e7f.slice/crio-223d7f711dd6fc5f01c2ca05c62b532e96d99b4629394e354436997906d310f4 WatchSource:0}: Error finding container 223d7f711dd6fc5f01c2ca05c62b532e96d99b4629394e354436997906d310f4: Status 404 returned error can't find the container with id 223d7f711dd6fc5f01c2ca05c62b532e96d99b4629394e354436997906d310f4
Nov 03 20:53:22.110730 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.110709    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 20:53:22.112213 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.112183    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:22.112294 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.112244    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:22.112294 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.112272    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:22.116389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.116372    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.147561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.147535    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:22.147950 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.147932    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:22.148866 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.148847    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 20:53:22.165399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.165358    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:53:22.165585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.165460    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:53:22.165585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.165529    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dddj5\""
Nov 03 20:53:22.165821 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.165789    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:53:22.176279 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.176050    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 20:53:22.216435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216394    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216622 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216453    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216622 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216497    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216610    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216669    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216862 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216776    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2df\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.216968 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.216946    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.317929 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.317892    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318000    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318027    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318090    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318261 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318135    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318261 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318151    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318369 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318328    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2df\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.318813 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.318786    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.320626 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.320600    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.320733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.320646    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.320811 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.320791    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.320916 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.320895    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.376955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.376874    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.383671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.383639    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2df\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kmpq4\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.436103 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.436073    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:22.644837 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.644745    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerStarted","Data":"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"}
Nov 03 20:53:22.644837 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.644810    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerStarted","Data":"223d7f711dd6fc5f01c2ca05c62b532e96d99b4629394e354436997906d310f4"}
Nov 03 20:53:22.762248 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.762187    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podStartSLOduration=1.762168554 podStartE2EDuration="1.762168554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:22.760789236 +0000 UTC m=+1427.687359475" watchObservedRunningTime="2026-11-03 20:53:22.762168554 +0000 UTC m=+1427.688738792"
Nov 03 20:53:22.873669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.873634    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 20:53:22.873792 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:22.873712    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780bf404_4247_4e97_8c62_bec19bb76e2e.slice/crio-0c171e2d3c370a1713783d986de04effd6dd9162cb6f764fe21e843ea54b5bca WatchSource:0}: Error finding container 0c171e2d3c370a1713783d986de04effd6dd9162cb6f764fe21e843ea54b5bca: Status 404 returned error can't find the container with id 0c171e2d3c370a1713783d986de04effd6dd9162cb6f764fe21e843ea54b5bca
Nov 03 20:53:22.877108 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.877073    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:22.877236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:22.877146    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:23.584971 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.584943    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:23.591389 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.591366    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:23.653720 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.653681    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" event={"ID":"780bf404-4247-4e97-8c62-bec19bb76e2e","Type":"ContainerStarted","Data":"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"}
Nov 03 20:53:23.654788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.654429    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:23.654788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.654470    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" event={"ID":"780bf404-4247-4e97-8c62-bec19bb76e2e","Type":"ContainerStarted","Data":"0c171e2d3c370a1713783d986de04effd6dd9162cb6f764fe21e843ea54b5bca"}
Nov 03 20:53:23.654788 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.654497    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:23.655204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.655185    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:53:23.655733 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.655711    4274 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kmpq4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:53:23.655909 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.655867    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:53:23.833110 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:23.832061    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" podStartSLOduration=1.832020217 podStartE2EDuration="1.832020217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:23.831432684 +0000 UTC m=+1428.758002925" watchObservedRunningTime="2026-11-03 20:53:23.832020217 +0000 UTC m=+1428.758590455"
Nov 03 20:53:24.704230 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:24.704190    4274 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kmpq4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:53:24.704812 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:24.704777    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:53:25.185061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.179234    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:53:25.316067 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.314463    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.379063 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.376365    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.382076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.381500    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.431069 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.427155    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.622090 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.618738    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.676081 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.672568    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.725448 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.725418    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 20:53:25.774960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.774918    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 20:53:25.781493 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.781452    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:25.830593 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:25.830533    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.192059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.184197    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.356060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.314614    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.356060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.317316    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:53:26.356060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.326662    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.356060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.326787    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.378068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.378019    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.407060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.378648    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:53:26.436170 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.436128    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.436740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.436716    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:53:26.543645 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.543607    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.662057 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.651088    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.671449 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.671411    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.689075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.678922    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:53:26.689075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.679696    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 20:53:26.689075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.679772    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.689075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.685973    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:53:26.769058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.768330    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.797122 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.792153    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:53:26.812088 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.799218    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:53:26.812088 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.799292    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:53:26.840206 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:26.821284    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:53:27.194301 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.194202    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:27.226004 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.225968    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:27.240221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.240191    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:53:27.324703 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.322583    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:27.418056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.378505    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:53:27.570066 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.562464    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:27.728071 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:53:27.727906    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:53:27.854213 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:27.791904    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 20:53:28.077060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.062236    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:28.198023 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.197945    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 20:53:28.198297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.198277    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:53:28.198463 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.198448    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:53:28.205136 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.205113    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:53:28.205322 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.205308    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 20:53:28.211163 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.211140    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 20:53:28.211354 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.211338    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 20:53:28.211470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.211456    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:53:28.241075 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.206862    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.242423 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.242400    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 20:53:28.242607 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.242591    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:53:28.242935 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.242913    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.335965 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.335933    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gmn\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336079    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336122    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978mx\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336158    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336404 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336193    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336404 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336228    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336508 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336402    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336508 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336455    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336612 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336524    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336648    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336713    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336810 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336760    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336872 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336857    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.336940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336914    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.336996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.336948    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.337120 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.337097    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.337202 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.337159    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.439201 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439158    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.439386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439226    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.439386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439256    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978mx\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.439386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439291    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.439386 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439344    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.439804 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439774    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.439970 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439939    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.439947    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440118 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440104    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440241 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440154    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440241 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440155    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440241 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440222    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440299    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440348    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440384    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440451    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440474    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440499    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440528    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440620 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440602    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440654    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.440881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440727    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gmn\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440739    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.440881 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.440748    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.441214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.441192    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.441638 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.441615    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.441778 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.441749    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.442055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.442006    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.443879 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.443857    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.444442 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.444407    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.445525 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.445494    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.446611 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.446587    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.537007 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.536970    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gmn\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.537759 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.537729    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.546235 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.545613    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.546235 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.546006    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978mx\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:28.596963 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.596927    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:28.612225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:28.612197    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:29.064723 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.064526    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 20:53:29.068479 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.068457    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:53:29.071203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.071163    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.071307 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.071241    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.071307 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.071286    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.083076 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:29.082983    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2686e8e2_1901_46fd_a36b_df2ada76ed59.slice/crio-d4b716d0a2667ef9767671464c6894bff1fbbd19e354aab31033c82b58e6f234 WatchSource:0}: Error finding container d4b716d0a2667ef9767671464c6894bff1fbbd19e354aab31033c82b58e6f234: Status 404 returned error can't find the container with id d4b716d0a2667ef9767671464c6894bff1fbbd19e354aab31033c82b58e6f234
Nov 03 20:53:29.088298 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.088260    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.088481 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.088452    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.088586 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.088507    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:29.249059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.248320    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" event={"ID":"f8969662-12c2-4849-b8a6-4993f5f760ca","Type":"ContainerStarted","Data":"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"}
Nov 03 20:53:29.249059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.248394    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" event={"ID":"f8969662-12c2-4849-b8a6-4993f5f760ca","Type":"ContainerStarted","Data":"c63b1fc3b2cf46e3fe8708d8c3e7e8097ac52059c76000234f8a307232ec6005"}
Nov 03 20:53:29.250546 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.249927    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" event={"ID":"2686e8e2-1901-46fd-a36b-df2ada76ed59","Type":"ContainerStarted","Data":"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"}
Nov 03 20:53:29.250546 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.249970    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" event={"ID":"2686e8e2-1901-46fd-a36b-df2ada76ed59","Type":"ContainerStarted","Data":"d4b716d0a2667ef9767671464c6894bff1fbbd19e354aab31033c82b58e6f234"}
Nov 03 20:53:29.597330 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.597297    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:29.616147 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.613554    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podStartSLOduration=2.613531815 podStartE2EDuration="2.613531815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:29.604095667 +0000 UTC m=+1434.530665905" watchObservedRunningTime="2026-11-03 20:53:29.613531815 +0000 UTC m=+1434.540102069"
Nov 03 20:53:29.616147 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.614500    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:29.619253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:29.619184    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podStartSLOduration=3.619166409 podStartE2EDuration="3.619166409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:29.490731112 +0000 UTC m=+1434.417301354" watchObservedRunningTime="2026-11-03 20:53:29.619166409 +0000 UTC m=+1434.545736710"
Nov 03 20:53:30.696053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:30.683919    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:30.696053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:30.683997    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:30.696053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:30.684309    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:30.696053 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:30.684339    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:31.604060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:31.600918    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:31.604060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:31.600979    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:31.613301 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:31.613269    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:31.613569 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:31.613524    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:32.600446 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:32.600409    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:32.601003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:32.600972    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:32.613473 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:32.613446    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:32.613615 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:32.613507    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:33.597904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:33.597418    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:33.597904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:33.597488    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:33.613473 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:33.613437    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:33.613845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:33.613514    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:34.601959 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:34.601917    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:34.602174 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:34.602000    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:34.613357 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:34.613308    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:34.613478 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:34.613377    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:35.598128 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:35.598089    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:53:35.598631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:35.598158    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 20:53:35.613215 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:35.613087    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:53:35.613215 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:35.613153    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:53:37.185850 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.185811    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:37.191076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.188219    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:37.191585 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.191537    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 20:53:37.193209 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.193188    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:37.195512 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.195489    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:37.196556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:37.196537    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:53:46.139468 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.139435    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 20:53:46.143952 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.143923    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.192402 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.192375    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\""
Nov 03 20:53:46.207364 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207340    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207402    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207502 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207467    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207502 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207496    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207520    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207681    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207745 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207727    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl6l\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207868 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207856    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.207903 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.207891    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.308709 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308677    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.308887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308728    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.308887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308769    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.308887 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308805    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309125 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308902    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309125 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.308936    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl6l\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309243 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309125    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309243 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309196    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309263    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309292    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309320    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309565 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309537    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309697 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309586    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.309697 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.309623    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.311607 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.311586    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.312238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.312221    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.350637 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.350588    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 20:53:46.382595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.382552    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl6l\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.383507 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.383475    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-v24v5\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.456831 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.456751    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:46.894057 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.894009    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 20:53:46.916094 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:53:46.916062    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2689076e_c8a2_49fb_8302_6576a600ba29.slice/crio-92025ad0d873f0cc38867c846ee50b6541b6d86d9791663c0631e0185d4d3f25 WatchSource:0}: Error finding container 92025ad0d873f0cc38867c846ee50b6541b6d86d9791663c0631e0185d4d3f25: Status 404 returned error can't find the container with id 92025ad0d873f0cc38867c846ee50b6541b6d86d9791663c0631e0185d4d3f25
Nov 03 20:53:46.918577 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.918538    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:46.918665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.918610    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:46.918665 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:46.918641    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:47.514516 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:47.514481    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" event={"ID":"2689076e-c8a2-49fb-8302-6576a600ba29","Type":"ContainerStarted","Data":"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"}
Nov 03 20:53:47.514932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:47.514536    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" event={"ID":"2689076e-c8a2-49fb-8302-6576a600ba29","Type":"ContainerStarted","Data":"92025ad0d873f0cc38867c846ee50b6541b6d86d9791663c0631e0185d4d3f25"}
Nov 03 20:53:47.733021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:47.732950    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podStartSLOduration=1.732930031 podStartE2EDuration="1.732930031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:47.681424858 +0000 UTC m=+1452.607995097" watchObservedRunningTime="2026-11-03 20:53:47.732930031 +0000 UTC m=+1452.659500269"
Nov 03 20:53:48.463288 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:48.462733    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:48.468056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:48.467142    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:48.468056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:48.467215    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:49.460085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:49.457391    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:49.460085 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:49.457455    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:50.466058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:50.465453    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:50.466058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:50.465520    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:51.458382 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:51.458343    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:51.458669 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:51.458634    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:52.460059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:52.457354    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:52.460059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:52.457416    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:53.468060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:53.467065    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:53.468060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:53.467124    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:54.457883 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:54.457843    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:53:54.458148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:54.457911    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 20:53:56.234335 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:56.234304    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:56.234723 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:56.234614    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:56.235525 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:56.235504    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 20:53:57.369469 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:53:57.369425    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:53:59.271177 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.271121    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.284190 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.284145    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.298150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.298105    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.313143 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.313105    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.571139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.568644    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.571139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.568982    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:53:59.571139 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.570527    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.635202 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.635164    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.661465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.661108    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.748919 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.748136    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.830269 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.830124    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:53:59.949060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:53:59.917939    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:54:00.027059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.026718    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.140418 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.140316    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:54:00.240094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.239130    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.302238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.300136    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.302238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.300608    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:54:00.302238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.302166    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p" podUID="2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:54:00.306232 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.303568    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:54:00.325062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.313669    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.325062 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.314156    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:54:00.353606 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.345898    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p"
Nov 03 20:54:00.386068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.385549    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 20:54:00.444930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:00.444654    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 20:54:23.667989 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.667917    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:54:23.672240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.672220    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.720932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.720914    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d9a2c67d-serving-cert\""
Nov 03 20:54:23.739076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.733012    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqm9g\" (UniqueName: \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.739076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.733363    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.739076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.733405    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.739076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.733436    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.771544 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.771517    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:54:23.835187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.835157    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqm9g\" (UniqueName: \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.835353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.835338    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.835400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.835376    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.835444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.835406    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.835521 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:54:23.835498    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-serving-cert: secret "maas-api-e2e-mt-d9a2c67d-serving-cert" not found
Nov 03 20:54:23.835597 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:54:23.835587    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls podName:f9c1357d-eb4a-4045-a942-4edff908f0a1 nodeName:}" failed. No retries permitted until 2026-11-03 20:54:24.335565768 +0000 UTC m=+1489.262135989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls") pod "maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" (UID: "f9c1357d-eb4a-4045-a942-4edff908f0a1") : secret "maas-api-e2e-mt-d9a2c67d-serving-cert" not found
Nov 03 20:54:23.836112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.836090    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.837775 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.837756    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:23.893138 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:23.893113    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqm9g\" (UniqueName: \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:24.345851 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:24.345821    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:24.348713 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:24.348685    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") pod \"maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:24.586982 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:24.586947    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:24.961416 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:24.961387    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:54:24.963630 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:54:24.963598    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c1357d_eb4a_4045_a942_4edff908f0a1.slice/crio-49fcaaad3bb0c53d7edc127b85a7ccac97e857c0b64e61c327275d01c1132ac7 WatchSource:0}: Error finding container 49fcaaad3bb0c53d7edc127b85a7ccac97e857c0b64e61c327275d01c1132ac7: Status 404 returned error can't find the container with id 49fcaaad3bb0c53d7edc127b85a7ccac97e857c0b64e61c327275d01c1132ac7
Nov 03 20:54:25.180801 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:25.180770    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" event={"ID":"f9c1357d-eb4a-4045-a942-4edff908f0a1","Type":"ContainerStarted","Data":"49fcaaad3bb0c53d7edc127b85a7ccac97e857c0b64e61c327275d01c1132ac7"}
Nov 03 20:54:26.188595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:26.188510    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" event={"ID":"f9c1357d-eb4a-4045-a942-4edff908f0a1","Type":"ContainerStarted","Data":"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"}
Nov 03 20:54:26.188595 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:26.188594    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:26.299379 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:26.299309    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" podStartSLOduration=2.466743555 podStartE2EDuration="3.29929162s" totalImagesPullingTime="832.548065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:26.298233219 +0000 UTC m=+1491.224803458" watchObservedRunningTime="2026-11-03 20:54:26.29929162 +0000 UTC m=+1491.225861859"
Nov 03 20:54:27.366880 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:54:27.366844    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:54:32.200150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:32.200121    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:54:46.643156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.643122    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 20:54:46.648825 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.648801    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.689322 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.689283    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"]
Nov 03 20:54:46.693598 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.693577    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.728787 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.728752    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.728909 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.728891    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.728968 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.728924    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729143    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729196    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729328 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729310    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnglz\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729376 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729366    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729486 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729465    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.729538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.729494    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.737819 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.737795    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 20:54:46.804690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.804655    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"]
Nov 03 20:54:46.831510 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831471    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831532    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831663 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831617    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.831741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831670    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnglz\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831741 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831725    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be93de48-76dc-499d-905d-7d7585cde6ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.831807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831755    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.831807 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831792    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831871 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831820    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.831871 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831863    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831886    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831932 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831902    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.831998 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.831931    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.832336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832315    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.832602 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832583    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.832676 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832660    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.832832 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832816    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.832933 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832918    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.832967 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.832955    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.833018 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.833003    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.833094 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.833077    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be93de48-76dc-499d-905d-7d7585cde6ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.833137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.833084    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.833137 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.833130    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gmz\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-kube-api-access-m5gmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.833368 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.833348    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.836013 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.835990    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.837400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.837380    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.889402 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.889374    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.889557 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.889491    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnglz\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-8z865\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.934080 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.933991    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934193 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934090    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be93de48-76dc-499d-905d-7d7585cde6ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934193 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934111    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934193 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934137    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934195    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934305 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934225    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934445 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934395    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934445 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934426    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be93de48-76dc-499d-905d-7d7585cde6ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934557 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934471    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gmz\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-kube-api-access-m5gmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934624 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934594    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934681 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934625    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.934743 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.934690    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.935149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.935125    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.935149 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.935139    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be93de48-76dc-499d-905d-7d7585cde6ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.936275 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.936258    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be93de48-76dc-499d-905d-7d7585cde6ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.936502 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.936477    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be93de48-76dc-499d-905d-7d7585cde6ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.962683 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.962652    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:46.990924 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.990896    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:46.991439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:46.991417    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gmz\" (UniqueName: \"kubernetes.io/projected/be93de48-76dc-499d-905d-7d7585cde6ae-kube-api-access-m5gmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-sfss5\" (UID: \"be93de48-76dc-499d-905d-7d7585cde6ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:47.008006 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.007985    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:47.394492 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.390716    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 20:54:47.400364 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.400314    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.400612 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.400587    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.400786 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.400769    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.432385 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.432358    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"]
Nov 03 20:54:47.441051 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:54:47.441004    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe93de48_76dc_499d_905d_7d7585cde6ae.slice/crio-9db4ec873a43fc79154420d9c29656e3d05a4a17ea81b6f8fbb361d2d7dd5c6b WatchSource:0}: Error finding container 9db4ec873a43fc79154420d9c29656e3d05a4a17ea81b6f8fbb361d2d7dd5c6b: Status 404 returned error can't find the container with id 9db4ec873a43fc79154420d9c29656e3d05a4a17ea81b6f8fbb361d2d7dd5c6b
Nov 03 20:54:47.444054 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.443991    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.444154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.444096    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.444154 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:47.444127    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:48.341048 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.340995    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" event={"ID":"be93de48-76dc-499d-905d-7d7585cde6ae","Type":"ContainerStarted","Data":"8c4a264b83bcda51340e1e364327c4aeefb0f5b669b88e1604f6c6d4eadcd528"}
Nov 03 20:54:48.341573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.341070    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" event={"ID":"be93de48-76dc-499d-905d-7d7585cde6ae","Type":"ContainerStarted","Data":"9db4ec873a43fc79154420d9c29656e3d05a4a17ea81b6f8fbb361d2d7dd5c6b"}
Nov 03 20:54:48.342844 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.342815    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" event={"ID":"e4f122ef-09d2-43d8-a65a-d1389a60964e","Type":"ContainerStarted","Data":"7d265ab0ea97960bbe7e55a22db6ea553a7d95cf014c279b7d7d4aabd45f9b00"}
Nov 03 20:54:48.342940 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.342865    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" event={"ID":"e4f122ef-09d2-43d8-a65a-d1389a60964e","Type":"ContainerStarted","Data":"c9e0ff29d4955dae7f056276c844878e9ae55180cd1248434e17c40af4264ca2"}
Nov 03 20:54:48.466070 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.459143    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podStartSLOduration=2.459123009 podStartE2EDuration="2.459123009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:48.457534547 +0000 UTC m=+1513.384104786" watchObservedRunningTime="2026-11-03 20:54:48.459123009 +0000 UTC m=+1513.385693248"
Nov 03 20:54:48.586059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.585673    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podStartSLOduration=2.585652846 podStartE2EDuration="2.585652846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:48.579694496 +0000 UTC m=+1513.506264735" watchObservedRunningTime="2026-11-03 20:54:48.585652846 +0000 UTC m=+1513.512223084"
Nov 03 20:54:48.967061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.964279    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:48.986060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.975181    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:48.986060 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:48.975261    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:49.009204 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:49.009172    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:49.010478 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:49.010454    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:49.010607 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:49.010500    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:49.974059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:49.973400    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:49.974059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:49.973461    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:50.008504 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:50.008468    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:50.008656 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:50.008521    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:50.963430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:50.963392    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:50.963690 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:50.963662    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:51.008591 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:51.008557    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:51.009024 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:51.008599    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:51.965433 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:51.965394    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:51.966404 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:51.966371    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:52.009277 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:52.009248    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:52.009731 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:52.009699    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:52.973489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:52.973444    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:52.973695 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:52.973510    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:53.012082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:53.009442    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:53.012082 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:53.009505    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:53.963160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:53.963117    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:53.963366 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:53.963174    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:54.009190 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:54.009158    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:54.009355 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:54.009208    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:54.963250 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:54.963214    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-8z865 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:54:54.963868 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:54.963264    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 03 20:54:55.008951 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:55.008909    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:54:55.009116 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:55.008966    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 20:54:56.415271 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.415193    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:56.416074 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.415947    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:56.417304 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.417283    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 20:54:56.443229 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.443201    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:56.444449 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.444428    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:56.445460 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:54:56.445440    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5"
Nov 03 20:54:57.369889 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:54:57.369851    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:55:12.596274 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:12.596238    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:55:12.711211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:12.711166    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:55:17.597308 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.597233    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" podUID="34954f14-83bc-4774-b11e-17042454fa06" containerName="payload-processing" containerID="cri-o://0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188" gracePeriod=25
Nov 03 20:55:17.712577 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.712540    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" podUID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" containerName="payload-pre-processing" containerID="cri-o://b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38" gracePeriod=25
Nov 03 20:55:17.898789 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.898764    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:55:17.986955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.985994    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\" (UniqueName: \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\") pod \"34954f14-83bc-4774-b11e-17042454fa06\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") "
Nov 03 20:55:17.986955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.986105    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\") pod \"34954f14-83bc-4774-b11e-17042454fa06\" (UID: \"34954f14-83bc-4774-b11e-17042454fa06\") "
Nov 03 20:55:17.986955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.986700    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume" pod "34954f14-83bc-4774-b11e-17042454fa06" (UID: "34954f14-83bc-4774-b11e-17042454fa06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:17.988754 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.988732    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m" pod "34954f14-83bc-4774-b11e-17042454fa06" (UID: "34954f14-83bc-4774-b11e-17042454fa06"). InnerVolumeSpecName "kube-api-access-48j2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:17.997799 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:17.997777    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:55:18.087543 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.087519    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\" (UniqueName: \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\") pod \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") "
Nov 03 20:55:18.087666 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.087655    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\") pod \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\" (UID: \"7320a3ea-f736-41c0-bfbc-e05413eb5eb9\") "
Nov 03 20:55:18.087980 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.087957    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume" pod "7320a3ea-f736-41c0-bfbc-e05413eb5eb9" (UID: "7320a3ea-f736-41c0-bfbc-e05413eb5eb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:18.088352 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.088326    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34954f14-83bc-4774-b11e-17042454fa06-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:18.088433 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.088359    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:18.088433 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.088374    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48j2m\" (UniqueName: \"kubernetes.io/projected/34954f14-83bc-4774-b11e-17042454fa06-kube-api-access-48j2m\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:18.090309 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.090288    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj" pod "7320a3ea-f736-41c0-bfbc-e05413eb5eb9" (UID: "7320a3ea-f736-41c0-bfbc-e05413eb5eb9"). InnerVolumeSpecName "kube-api-access-vbcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:18.190369 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.190303    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcfj\" (UniqueName: \"kubernetes.io/projected/7320a3ea-f736-41c0-bfbc-e05413eb5eb9-kube-api-access-vbcfj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:18.641629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.641600    4274 generic.go:356] "Generic (PLEG): container finished" podID="34954f14-83bc-4774-b11e-17042454fa06" containerID="0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188" exitCode=0
Nov 03 20:55:18.641996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.641644    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"
Nov 03 20:55:18.641996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.641677    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" event={"ID":"34954f14-83bc-4774-b11e-17042454fa06","Type":"ContainerDied","Data":"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"}
Nov 03 20:55:18.641996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.641721    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk" event={"ID":"34954f14-83bc-4774-b11e-17042454fa06","Type":"ContainerDied","Data":"2ce3659e549358b521bde3fbb9836d7189497bbba89100adebb35b55b2a2f374"}
Nov 03 20:55:18.641996 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.641745    4274 scope.go:122] "RemoveContainer" containerID="0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"
Nov 03 20:55:18.643383 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.643352    4274 generic.go:356] "Generic (PLEG): container finished" podID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" containerID="b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38" exitCode=0
Nov 03 20:55:18.643491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.643377    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" event={"ID":"7320a3ea-f736-41c0-bfbc-e05413eb5eb9","Type":"ContainerDied","Data":"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"}
Nov 03 20:55:18.643491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.643394    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"
Nov 03 20:55:18.643491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.643426    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v" event={"ID":"7320a3ea-f736-41c0-bfbc-e05413eb5eb9","Type":"ContainerDied","Data":"85fd0638aaf6c7979d74b09e5cfb922ce13c3d15a65dde6c4746edc0e226fd45"}
Nov 03 20:55:18.645504 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.645483    4274 generic.go:356] "Generic (PLEG): container finished" podID="93136739-ecc0-4939-a91b-fab66a741e7f" containerID="fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64" exitCode=0
Nov 03 20:55:18.645606 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.645517    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerDied","Data":"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"}
Nov 03 20:55:18.645948 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.645931    4274 scope.go:122] "RemoveContainer" containerID="fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"
Nov 03 20:55:18.647536 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.647508    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:18.647635 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.647572    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:18.647635 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.647627    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:18.652530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.652510    4274 scope.go:122] "RemoveContainer" containerID="0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"
Nov 03 20:55:18.652857 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:18.652821    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188\": container with ID starting with 0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188 not found: ID does not exist" containerID="0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"
Nov 03 20:55:18.652952 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.652855    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188"} err="failed to get container status \"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188\": rpc error: code = NotFound desc = could not find container \"0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188\": container with ID starting with 0d9ab911f00020766ecc8779bdfb8561ce8efd23b8a4e6c35c67a2d028171188 not found: ID does not exist"
Nov 03 20:55:18.652952 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.652877    4274 scope.go:122] "RemoveContainer" containerID="b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"
Nov 03 20:55:18.663336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.663319    4274 scope.go:122] "RemoveContainer" containerID="b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"
Nov 03 20:55:18.663575 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:18.663554    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38\": container with ID starting with b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38 not found: ID does not exist" containerID="b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"
Nov 03 20:55:18.663634 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.663583    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38"} err="failed to get container status \"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38\": rpc error: code = NotFound desc = could not find container \"b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38\": container with ID starting with b6c5b7844128218fbd195f388f754c69b47f5906198fcb9274633fca77ce7e38 not found: ID does not exist"
Nov 03 20:55:18.797962 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.797927    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:55:18.802623 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:18.802591    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15564eb1-5c4d4577c9-nfr8v"]
Nov 03 20:55:19.019476 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:19.019446    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:55:19.025029 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:19.025002    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15564eb1-7756c6ddcc-gxrvk"]
Nov 03 20:55:19.214792 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:19.214757    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34954f14-83bc-4774-b11e-17042454fa06" path="/var/lib/kubelet/pods/34954f14-83bc-4774-b11e-17042454fa06/volumes"
Nov 03 20:55:19.215235 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:19.215219    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" path="/var/lib/kubelet/pods/7320a3ea-f736-41c0-bfbc-e05413eb5eb9/volumes"
Nov 03 20:55:19.677150 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:19.677109    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerStarted","Data":"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"}
Nov 03 20:55:20.585171 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:20.585131    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:55:20.586189 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:20.586158    4274 patch_prober.go:29] interesting pod/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:55:20.586327 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:20.586231    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 20:55:21.585431 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:21.585398    4274 patch_prober.go:29] interesting pod/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:55:21.586160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:21.586074    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 20:55:21.923396 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:21.923312    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:55:22.044367 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:22.044334    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:55:22.584425 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:22.584389    4274 patch_prober.go:29] interesting pod/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:55:22.584601 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:22.584452    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 20:55:23.584458 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:23.584426    4274 patch_prober.go:29] interesting pod/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:55:23.584853 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:23.584475    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 20:55:23.661342 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:23.661313    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:55:24.584216 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:24.584186    4274 patch_prober.go:29] interesting pod/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:55:24.584412 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:24.584233    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 20:55:25.588710 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:25.588683    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:55:25.589835 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:25.589817    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:55:26.924567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:26.924519    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" podUID="e6355981-58fa-4b5a-ad69-82a5a595de73" containerName="payload-processing" containerID="cri-o://fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9" gracePeriod=25
Nov 03 20:55:27.044824 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.044777    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" podUID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" containerName="payload-pre-processing" containerID="cri-o://10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c" gracePeriod=25
Nov 03 20:55:27.218197 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.218169    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:55:27.285806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.285026    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\") pod \"e6355981-58fa-4b5a-ad69-82a5a595de73\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") "
Nov 03 20:55:27.285806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.285143    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\" (UniqueName: \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\") pod \"e6355981-58fa-4b5a-ad69-82a5a595de73\" (UID: \"e6355981-58fa-4b5a-ad69-82a5a595de73\") "
Nov 03 20:55:27.285806 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.285426    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume" pod "e6355981-58fa-4b5a-ad69-82a5a595de73" (UID: "e6355981-58fa-4b5a-ad69-82a5a595de73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:27.286134 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.285950    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6355981-58fa-4b5a-ad69-82a5a595de73-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:27.287706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.287673    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5" pod "e6355981-58fa-4b5a-ad69-82a5a595de73" (UID: "e6355981-58fa-4b5a-ad69-82a5a595de73"). InnerVolumeSpecName "kube-api-access-v8pn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:27.321675 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.321652    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:55:27.368804 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:27.368767    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:55:27.386485 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.386449    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\") pod \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") "
Nov 03 20:55:27.386873 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.386836    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume" pod "1c9d5973-2866-4183-92c1-0b3b2e930ab1" (UID: "1c9d5973-2866-4183-92c1-0b3b2e930ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:27.387147 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.387126    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\" (UniqueName: \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\") pod \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\" (UID: \"1c9d5973-2866-4183-92c1-0b3b2e930ab1\") "
Nov 03 20:55:27.387693 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.387671    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pn5\" (UniqueName: \"kubernetes.io/projected/e6355981-58fa-4b5a-ad69-82a5a595de73-kube-api-access-v8pn5\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:27.387793 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.387698    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9d5973-2866-4183-92c1-0b3b2e930ab1-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:27.396348 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.396318    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t" pod "1c9d5973-2866-4183-92c1-0b3b2e930ab1" (UID: "1c9d5973-2866-4183-92c1-0b3b2e930ab1"). InnerVolumeSpecName "kube-api-access-l487t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:27.489226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.489127    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l487t\" (UniqueName: \"kubernetes.io/projected/1c9d5973-2866-4183-92c1-0b3b2e930ab1-kube-api-access-l487t\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:27.735441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.735407    4274 generic.go:356] "Generic (PLEG): container finished" podID="e6355981-58fa-4b5a-ad69-82a5a595de73" containerID="fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9" exitCode=0
Nov 03 20:55:27.735629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.735456    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"
Nov 03 20:55:27.735629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.735496    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" event={"ID":"e6355981-58fa-4b5a-ad69-82a5a595de73","Type":"ContainerDied","Data":"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"}
Nov 03 20:55:27.735629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.735560    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5" event={"ID":"e6355981-58fa-4b5a-ad69-82a5a595de73","Type":"ContainerDied","Data":"910bb2cafdebb0da05b5d9a41c8f6ad64179e7363a346086777cb911f48e45dc"}
Nov 03 20:55:27.735629 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.735590    4274 scope.go:122] "RemoveContainer" containerID="fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"
Nov 03 20:55:27.736962 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.736941    4274 generic.go:356] "Generic (PLEG): container finished" podID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" containerID="10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c" exitCode=0
Nov 03 20:55:27.737104 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.736991    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"
Nov 03 20:55:27.737104 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.737051    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" event={"ID":"1c9d5973-2866-4183-92c1-0b3b2e930ab1","Type":"ContainerDied","Data":"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"}
Nov 03 20:55:27.737226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.737104    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z" event={"ID":"1c9d5973-2866-4183-92c1-0b3b2e930ab1","Type":"ContainerDied","Data":"2e38b45e42442055ef405edbf8289574a9a7fe678ce3d00e1ec3d2f6841358d5"}
Nov 03 20:55:27.746718 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.746700    4274 scope.go:122] "RemoveContainer" containerID="fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"
Nov 03 20:55:27.747011 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:27.746979    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9\": container with ID starting with fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9 not found: ID does not exist" containerID="fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"
Nov 03 20:55:27.747160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.747018    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9"} err="failed to get container status \"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9\": rpc error: code = NotFound desc = could not find container \"fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9\": container with ID starting with fe49babcb8c33c2c8ec3dd5b2059828ab555718fcb564e954705dfcc4e8042b9 not found: ID does not exist"
Nov 03 20:55:27.747160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.747053    4274 scope.go:122] "RemoveContainer" containerID="10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"
Nov 03 20:55:27.756706 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.756687    4274 scope.go:122] "RemoveContainer" containerID="10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"
Nov 03 20:55:27.756976 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:27.756956    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c\": container with ID starting with 10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c not found: ID does not exist" containerID="10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"
Nov 03 20:55:27.757059 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.756981    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c"} err="failed to get container status \"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c\": rpc error: code = NotFound desc = could not find container \"10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c\": container with ID starting with 10bd773112bd3a7e8811c65688b4df8fbe82a8e59dd07ccd7dfd4cddf6f3d85c not found: ID does not exist"
Nov 03 20:55:27.903573 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.903540    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:55:27.907646 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:27.907625    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9337cba-7f674b876c-9jh4z"]
Nov 03 20:55:28.010824 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:28.010795    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:55:28.015173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:28.015149    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9337cba-db8d4fc9-kz8n5"]
Nov 03 20:55:29.217766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:29.217732    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" path="/var/lib/kubelet/pods/1c9d5973-2866-4183-92c1-0b3b2e930ab1/volumes"
Nov 03 20:55:29.218269 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:55:29.218251    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6355981-58fa-4b5a-ad69-82a5a595de73" path="/var/lib/kubelet/pods/e6355981-58fa-4b5a-ad69-82a5a595de73/volumes"
Nov 03 20:55:57.384295 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:55:57.384249    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:56:27.366284 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:27.366250    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:56:35.280341 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280023    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:56:35.280915 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280901    4274 cpu_manager.go:414] "containerMap: removing container" podUID="34954f14-83bc-4774-b11e-17042454fa06" containerName="payload-processing"
Nov 03 20:56:35.280960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280916    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34954f14-83bc-4774-b11e-17042454fa06" containerName="payload-processing"
Nov 03 20:56:35.280960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280939    4274 cpu_manager.go:414] "containerMap: removing container" podUID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" containerName="payload-pre-processing"
Nov 03 20:56:35.280960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280945    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" containerName="payload-pre-processing"
Nov 03 20:56:35.280960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280957    4274 cpu_manager.go:414] "containerMap: removing container" podUID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" containerName="payload-pre-processing"
Nov 03 20:56:35.280960 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280962    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" containerName="payload-pre-processing"
Nov 03 20:56:35.281153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280978    4274 cpu_manager.go:414] "containerMap: removing container" podUID="e6355981-58fa-4b5a-ad69-82a5a595de73" containerName="payload-processing"
Nov 03 20:56:35.281153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.280985    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6355981-58fa-4b5a-ad69-82a5a595de73" containerName="payload-processing"
Nov 03 20:56:35.281153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.281149    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="34954f14-83bc-4774-b11e-17042454fa06" containerName="payload-processing"
Nov 03 20:56:35.281259 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.281164    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9d5973-2866-4183-92c1-0b3b2e930ab1" containerName="payload-pre-processing"
Nov 03 20:56:35.281259 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.281172    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="7320a3ea-f736-41c0-bfbc-e05413eb5eb9" containerName="payload-pre-processing"
Nov 03 20:56:35.281259 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.281180    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6355981-58fa-4b5a-ad69-82a5a595de73" containerName="payload-processing"
Nov 03 20:56:35.284790 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.284771    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.382240 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.382209    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-df4313ad-serving-cert\""
Nov 03 20:56:35.390444 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.390409    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:56:35.430055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.430003    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.430435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.430416    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.431163 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.431142    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.431393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.431374    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc4s\" (UniqueName: \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.558876 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.558781    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.573481 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.573450    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.573778 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.573750    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc4s\" (UniqueName: \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.574089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.574065    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.574958 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.574924    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.575117 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:35.559207    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-serving-cert: secret "maas-api-e2e-ait-df4313ad-serving-cert" not found
Nov 03 20:56:35.575117 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:35.575050    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls podName:690d8beb-ba5e-49d3-8929-c47aa32def2e nodeName:}" failed. No retries permitted until 2026-11-03 20:56:36.07501263 +0000 UTC m=+1621.001582851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls") pod "maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" (UID: "690d8beb-ba5e-49d3-8929-c47aa32def2e") : secret "maas-api-e2e-ait-df4313ad-serving-cert" not found
Nov 03 20:56:35.597879 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.597852    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:35.637671 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:35.634643    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc4s\" (UniqueName: \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:36.115947 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:36.115908    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:36.118633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:36.118605    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") pod \"maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:36.199159 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:36.199130    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:36.586271 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:56:36.585608    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690d8beb_ba5e_49d3_8929_c47aa32def2e.slice/crio-0ec672149d9d242bb659decec6bd24f9bdff2fe3131b7126bd3684013cab750a WatchSource:0}: Error finding container 0ec672149d9d242bb659decec6bd24f9bdff2fe3131b7126bd3684013cab750a: Status 404 returned error can't find the container with id 0ec672149d9d242bb659decec6bd24f9bdff2fe3131b7126bd3684013cab750a
Nov 03 20:56:36.586271 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:36.586171    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:56:37.313518 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:37.313479    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" event={"ID":"690d8beb-ba5e-49d3-8929-c47aa32def2e","Type":"ContainerStarted","Data":"0ec672149d9d242bb659decec6bd24f9bdff2fe3131b7126bd3684013cab750a"}
Nov 03 20:56:38.347855 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:38.347809    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" event={"ID":"690d8beb-ba5e-49d3-8929-c47aa32def2e","Type":"ContainerStarted","Data":"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"}
Nov 03 20:56:38.348834 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:38.348809    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:38.473770 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:38.473695    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" podStartSLOduration=2.495563237 podStartE2EDuration="3.473675084s" totalImagesPullingTime="978.111847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:38.473494423 +0000 UTC m=+1623.400064661" watchObservedRunningTime="2026-11-03 20:56:38.473675084 +0000 UTC m=+1623.400245321"
Nov 03 20:56:40.408991 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.408951    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:56:40.421339 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.420284    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.445628 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.445591    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.445823 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.445654    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.445823 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.445721    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7658\" (UniqueName: \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.445823 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.445765    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.467586 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.467549    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:56:40.508001 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.507024    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-299a741b-serving-cert\""
Nov 03 20:56:40.550694 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.550630    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7658\" (UniqueName: \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.550901 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.550885    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.551203 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.551188    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.551300 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.551290    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.552420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.552400    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.552773 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:40.552756    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-serving-cert: secret "maas-api-e2e-derive-299a741b-serving-cert" not found
Nov 03 20:56:40.552922 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:40.552913    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls podName:d3c95f2a-29da-40c4-8d06-e089689134dd nodeName:}" failed. No retries permitted until 2026-11-03 20:56:41.052897267 +0000 UTC m=+1625.979467483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls") pod "maas-api-e2e-derive-299a741b-54b9759995-mdh7n" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd") : secret "maas-api-e2e-derive-299a741b-serving-cert" not found
Nov 03 20:56:40.555859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.555781    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:40.612224 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:40.612195    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7658\" (UniqueName: \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:41.059454 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:41.059415    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:41.059648 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:41.059560    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-serving-cert: secret "maas-api-e2e-derive-299a741b-serving-cert" not found
Nov 03 20:56:41.059648 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:41.059642    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls podName:d3c95f2a-29da-40c4-8d06-e089689134dd nodeName:}" failed. No retries permitted until 2026-11-03 20:56:42.059625554 +0000 UTC m=+1626.986195782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls") pod "maas-api-e2e-derive-299a741b-54b9759995-mdh7n" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd") : secret "maas-api-e2e-derive-299a741b-serving-cert" not found
Nov 03 20:56:42.074628 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:42.074594    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:42.077430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:42.077396    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"maas-api-e2e-derive-299a741b-54b9759995-mdh7n\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:42.237927 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:42.237897    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:42.614048 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:42.614008    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:56:42.614664 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:56:42.614636    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c95f2a_29da_40c4_8d06_e089689134dd.slice/crio-4a50fe492366998b3e80e818fe794f7f98b52a0d6965dc1a8109e22422bfc877 WatchSource:0}: Error finding container 4a50fe492366998b3e80e818fe794f7f98b52a0d6965dc1a8109e22422bfc877: Status 404 returned error can't find the container with id 4a50fe492366998b3e80e818fe794f7f98b52a0d6965dc1a8109e22422bfc877
Nov 03 20:56:43.441687 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:43.441648    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" event={"ID":"d3c95f2a-29da-40c4-8d06-e089689134dd","Type":"ContainerStarted","Data":"4a50fe492366998b3e80e818fe794f7f98b52a0d6965dc1a8109e22422bfc877"}
Nov 03 20:56:44.451003 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:44.450963    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" event={"ID":"d3c95f2a-29da-40c4-8d06-e089689134dd","Type":"ContainerStarted","Data":"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"}
Nov 03 20:56:44.451419 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:44.451090    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:44.606639 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:44.606568    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" podStartSLOduration=3.7527018 podStartE2EDuration="4.606548322s" totalImagesPullingTime="853.846522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:44.560385025 +0000 UTC m=+1629.486955261" watchObservedRunningTime="2026-11-03 20:56:44.606548322 +0000 UTC m=+1629.533118560"
Nov 03 20:56:45.422628 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:45.422588    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:56:50.464426 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:56:50.464396    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:56:57.366589 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:56:57.366545    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:57:27.365206 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:57:27.365170    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:57:42.949054 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:42.948962    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:57:42.949543 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:42.949250    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy" containerID="cri-o://37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915" gracePeriod=30
Nov 03 20:57:42.994955 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:42.994923    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:57:42.995153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:42.995136    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy" containerID="cri-o://ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58" gracePeriod=30
Nov 03 20:57:44.877724 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:44.877688    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:57:44.878187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:44.877885    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy" containerID="cri-o://4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b" gracePeriod=30
Nov 03 20:57:48.332834 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.332799    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:57:48.361814 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.361791    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:57:48.366165 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366143    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366289 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366178    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366393 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366374    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366401    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366420    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.366514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366475    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366506    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366499    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.366752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366541    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366642    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366710    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\") pod \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\" (UID: \"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c\") "
Nov 03 20:57:48.366752 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366723    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.366988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366912    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:48.366988 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.366918    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.367632 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.367607    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.367736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.367635    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.367736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.367652    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.367736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.367668    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.367736 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.367683    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.368699 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.368678    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.368941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.368915    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:48.369278 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.369252    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:48.369567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.369546    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk" pod "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" (UID: "46e0cdfe-2b8c-478d-8f57-1b22a95dd75c"). InnerVolumeSpecName "kube-api-access-mngsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:48.468808 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.468776    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.468987 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.468836    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.468987 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.468883    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.468987 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.468916    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.468987 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.468984    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.469264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469058    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.469264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469089    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.469264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469123    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.469264 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469226    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\") pod \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\" (UID: \"bdf6780e-68d8-4240-ac2f-79b03f05cf7c\") "
Nov 03 20:57:48.469474 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469322    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:48.469474 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469334    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.469584 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469560    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.469756 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469723    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.469756 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.469742    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.470338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470249    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470280    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470297    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470312    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470338 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470327    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470343    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470357    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngsk\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-kube-api-access-mngsk\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470371    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.470667 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.470384    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.471394 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.471364    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:48.471499 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.471407    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:48.471563 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.471504    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "kube-api-access-qsxjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:48.471563 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.471540    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token" pod "bdf6780e-68d8-4240-ac2f-79b03f05cf7c" (UID: "bdf6780e-68d8-4240-ac2f-79b03f05cf7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:48.571530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.571501    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxjg\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-kube-api-access-qsxjg\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.571530 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.571527    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.571726 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.571536    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.571726 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.571546    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6780e-68d8-4240-ac2f-79b03f05cf7c-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:48.881916 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.881829    4274 generic.go:356] "Generic (PLEG): container finished" podID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerID="37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915" exitCode=0
Nov 03 20:57:48.881916 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.881878    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"
Nov 03 20:57:48.882145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.881922    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" event={"ID":"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c","Type":"ContainerDied","Data":"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"}
Nov 03 20:57:48.882145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.882014    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw" event={"ID":"46e0cdfe-2b8c-478d-8f57-1b22a95dd75c","Type":"ContainerDied","Data":"f611c271200de9a67addbe93b78dabf1760848abf38f9bcab5ff6c9f15edfb4c"}
Nov 03 20:57:48.882145 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.882063    4274 scope.go:122] "RemoveContainer" containerID="37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"
Nov 03 20:57:48.883477 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.883452    4274 generic.go:356] "Generic (PLEG): container finished" podID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerID="ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58" exitCode=0
Nov 03 20:57:48.883633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.883484    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" event={"ID":"bdf6780e-68d8-4240-ac2f-79b03f05cf7c","Type":"ContainerDied","Data":"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"}
Nov 03 20:57:48.883633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.883501    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"
Nov 03 20:57:48.883633 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.883531    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v" event={"ID":"bdf6780e-68d8-4240-ac2f-79b03f05cf7c","Type":"ContainerDied","Data":"c4255f5128e79d07c5912bd59773a2ed76f11ad17fe737210b3333184ae4deab"}
Nov 03 20:57:48.892601 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.892586    4274 scope.go:122] "RemoveContainer" containerID="37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"
Nov 03 20:57:48.892948 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:57:48.892921    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915\": container with ID starting with 37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915 not found: ID does not exist" containerID="37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"
Nov 03 20:57:48.893061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.892957    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915"} err="failed to get container status \"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915\": rpc error: code = NotFound desc = could not find container \"37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915\": container with ID starting with 37f6153cdadaf9de52851743bac6933ae6df7e7bbc0f5fb177293de0bae8e915 not found: ID does not exist"
Nov 03 20:57:48.893061 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.892985    4274 scope.go:122] "RemoveContainer" containerID="ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"
Nov 03 20:57:48.905426 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.905407    4274 scope.go:122] "RemoveContainer" containerID="ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"
Nov 03 20:57:48.905703 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:57:48.905685    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58\": container with ID starting with ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58 not found: ID does not exist" containerID="ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"
Nov 03 20:57:48.905751 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:48.905709    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58"} err="failed to get container status \"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58\": rpc error: code = NotFound desc = could not find container \"ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58\": container with ID starting with ffebfb308152aed47aab99f987957bc241439f6f1ed72523716ec34b7d69ba58 not found: ID does not exist"
Nov 03 20:57:49.037920 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.037890    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:57:49.083921 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.083891    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-449cw"]
Nov 03 20:57:49.145173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.145102    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:57:49.151618 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.151596    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bsh7v"]
Nov 03 20:57:49.214863 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.214833    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" path="/var/lib/kubelet/pods/46e0cdfe-2b8c-478d-8f57-1b22a95dd75c/volumes"
Nov 03 20:57:49.215400 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:49.215385    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" path="/var/lib/kubelet/pods/bdf6780e-68d8-4240-ac2f-79b03f05cf7c/volumes"
Nov 03 20:57:50.153773 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.153719    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:57:50.194296 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194269    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194304    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194321    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194347    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194435 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194409    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194455    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194568    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:50.194658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194597    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194608    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:50.194913 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194687    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.194913 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194825    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:50.194913 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194840    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\") pod \"6644bca6-51e3-426d-9e37-8dea63ba02aa\" (UID: \"6644bca6-51e3-426d-9e37-8dea63ba02aa\") "
Nov 03 20:57:50.195127 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.194945    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:50.195187 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195172    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:50.195508 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195487    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.195580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195513    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.195580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195528    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.195580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195542    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.195580 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.195557    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6644bca6-51e3-426d-9e37-8dea63ba02aa-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.196415 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.196398    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:50.196658 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.196639    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "kube-api-access-b7btc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:50.196975 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.196957    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:50.197021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.197003    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token" pod "6644bca6-51e3-426d-9e37-8dea63ba02aa" (UID: "6644bca6-51e3-426d-9e37-8dea63ba02aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:50.297319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.297286    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.297319 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.297323    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7btc\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-kube-api-access-b7btc\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.297533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.297343    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.297533 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.297362    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6644bca6-51e3-426d-9e37-8dea63ba02aa-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:50.902012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.901969    4274 generic.go:356] "Generic (PLEG): container finished" podID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerID="4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b" exitCode=0
Nov 03 20:57:50.902211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.902022    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"
Nov 03 20:57:50.902211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.902061    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" event={"ID":"6644bca6-51e3-426d-9e37-8dea63ba02aa","Type":"ContainerDied","Data":"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"}
Nov 03 20:57:50.902211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.902116    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k" event={"ID":"6644bca6-51e3-426d-9e37-8dea63ba02aa","Type":"ContainerDied","Data":"8558ccea503f8b261af80ba383f410d5f33895f31431f101268527724479bdae"}
Nov 03 20:57:50.902211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.902139    4274 scope.go:122] "RemoveContainer" containerID="4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"
Nov 03 20:57:50.913423 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.913404    4274 scope.go:122] "RemoveContainer" containerID="4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"
Nov 03 20:57:50.913685 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:57:50.913666    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b\": container with ID starting with 4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b not found: ID does not exist" containerID="4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"
Nov 03 20:57:50.913751 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:50.913699    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b"} err="failed to get container status \"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b\": rpc error: code = NotFound desc = could not find container \"4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b\": container with ID starting with 4cf8f9360202c7c0644000043b0e4a66026e17b2e81a51745373c5b90fcb994b not found: ID does not exist"
Nov 03 20:57:51.060930 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:51.060894    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:57:51.106617 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:51.106587    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-hrd8k"]
Nov 03 20:57:51.214398 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:57:51.214320    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" path="/var/lib/kubelet/pods/6644bca6-51e3-426d-9e37-8dea63ba02aa/volumes"
Nov 03 20:57:57.366361 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:57:57.366326    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:58:01.692919 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.692878    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 20:58:01.694113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694085    4274 cpu_manager.go:414] "containerMap: removing container" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy"
Nov 03 20:58:01.694113 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694112    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694146    4274 cpu_manager.go:414] "containerMap: removing container" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694154    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694174    4274 cpu_manager.go:414] "containerMap: removing container" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694181    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694384    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e0cdfe-2b8c-478d-8f57-1b22a95dd75c" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694405    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf6780e-68d8-4240-ac2f-79b03f05cf7c" containerName="istio-proxy"
Nov 03 20:58:01.696898 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.694417    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6644bca6-51e3-426d-9e37-8dea63ba02aa" containerName="istio-proxy"
Nov 03 20:58:01.700443 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.700420    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.751049 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.751007    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5b12d6\""
Nov 03 20:58:01.772953 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.772844    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzd8v\" (UniqueName: \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.773382 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.773009    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.801798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.801761    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5b12d6-dockercfg-ptjlx\""
Nov 03 20:58:01.803284 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.803253    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 20:58:01.875057 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.874588    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzd8v\" (UniqueName: \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.875057 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.874752    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.885067 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.876262    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:01.933578 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:01.933539    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzd8v\" (UniqueName: \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\") pod \"payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:02.017209 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:02.017182    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:02.402659 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:58:02.402620    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd2eba3_d917_4d84_8d6a_e4687354d8b5.slice/crio-c2c677f1f786e11d1dee860788d7aabb494068204e12ff1c86f73d60ccc51a03 WatchSource:0}: Error finding container c2c677f1f786e11d1dee860788d7aabb494068204e12ff1c86f73d60ccc51a03: Status 404 returned error can't find the container with id c2c677f1f786e11d1dee860788d7aabb494068204e12ff1c86f73d60ccc51a03
Nov 03 20:58:02.403138 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:02.403107    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 20:58:02.998348 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:02.998311    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" event={"ID":"4bd2eba3-d917-4d84-8d6a-e4687354d8b5","Type":"ContainerStarted","Data":"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"}
Nov 03 20:58:02.998768 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:02.998375    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" event={"ID":"4bd2eba3-d917-4d84-8d6a-e4687354d8b5","Type":"ContainerStarted","Data":"c2c677f1f786e11d1dee860788d7aabb494068204e12ff1c86f73d60ccc51a03"}
Nov 03 20:58:02.998768 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:02.998406    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:03.106860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:03.106813    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" podStartSLOduration=2.106799606 podStartE2EDuration="2.106799606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:03.106558237 +0000 UTC m=+1708.033128473" watchObservedRunningTime="2026-11-03 20:58:03.106799606 +0000 UTC m=+1708.033369842"
Nov 03 20:58:05.466923 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.466891    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:58:05.467664 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.467114    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" podUID="f9c1357d-eb4a-4045-a942-4edff908f0a1" containerName="maas-api" containerID="cri-o://d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5" gracePeriod=30
Nov 03 20:58:05.791372 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.791250    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:58:05.851789 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.851743    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\" (UniqueName: \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\") pod \"f9c1357d-eb4a-4045-a942-4edff908f0a1\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") "
Nov 03 20:58:05.851997 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.851973    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") pod \"f9c1357d-eb4a-4045-a942-4edff908f0a1\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") "
Nov 03 20:58:05.852144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.852119    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\") pod \"f9c1357d-eb4a-4045-a942-4edff908f0a1\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") "
Nov 03 20:58:05.852301 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.852273    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\") pod \"f9c1357d-eb4a-4045-a942-4edff908f0a1\" (UID: \"f9c1357d-eb4a-4045-a942-4edff908f0a1\") "
Nov 03 20:58:05.854047 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.853987    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca" pod "f9c1357d-eb4a-4045-a942-4edff908f0a1" (UID: "f9c1357d-eb4a-4045-a942-4edff908f0a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:05.855581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.855557    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g" pod "f9c1357d-eb4a-4045-a942-4edff908f0a1" (UID: "f9c1357d-eb4a-4045-a942-4edff908f0a1"). InnerVolumeSpecName "kube-api-access-kqm9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:05.862066 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.856464    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls" pod "f9c1357d-eb4a-4045-a942-4edff908f0a1" (UID: "f9c1357d-eb4a-4045-a942-4edff908f0a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:05.874238 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.874205    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs" pod "f9c1357d-eb4a-4045-a942-4edff908f0a1" (UID: "f9c1357d-eb4a-4045-a942-4edff908f0a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:05.955076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.954999    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqm9g\" (UniqueName: \"kubernetes.io/projected/f9c1357d-eb4a-4045-a942-4edff908f0a1-kube-api-access-kqm9g\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:05.955076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.955054    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:05.955076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.955070    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c1357d-eb4a-4045-a942-4edff908f0a1-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:05.955076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:05.955084    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c1357d-eb4a-4045-a942-4edff908f0a1-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:06.019759 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.019724    4274 generic.go:356] "Generic (PLEG): container finished" podID="f9c1357d-eb4a-4045-a942-4edff908f0a1" containerID="d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5" exitCode=0
Nov 03 20:58:06.019941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.019758    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" event={"ID":"f9c1357d-eb4a-4045-a942-4edff908f0a1","Type":"ContainerDied","Data":"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"}
Nov 03 20:58:06.019941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.019775    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"
Nov 03 20:58:06.019941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.019827    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw" event={"ID":"f9c1357d-eb4a-4045-a942-4edff908f0a1","Type":"ContainerDied","Data":"49fcaaad3bb0c53d7edc127b85a7ccac97e857c0b64e61c327275d01c1132ac7"}
Nov 03 20:58:06.019941 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.019853    4274 scope.go:122] "RemoveContainer" containerID="d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"
Nov 03 20:58:06.036297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.031339    4274 scope.go:122] "RemoveContainer" containerID="d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"
Nov 03 20:58:06.036297 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:06.031813    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5\": container with ID starting with d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5 not found: ID does not exist" containerID="d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"
Nov 03 20:58:06.036297 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.031849    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5"} err="failed to get container status \"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5\": rpc error: code = NotFound desc = could not find container \"d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5\": container with ID starting with d5a5be87a3a5300716f416721f44bd9250ea251fcb08a84c3af87f626a640ee5 not found: ID does not exist"
Nov 03 20:58:06.175722 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.175668    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:58:06.179423 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:06.179398    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d9a2c67d-67bf8cdb48-nn2vw"]
Nov 03 20:58:07.214845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:07.214812    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c1357d-eb4a-4045-a942-4edff908f0a1" path="/var/lib/kubelet/pods/f9c1357d-eb4a-4045-a942-4edff908f0a1/volumes"
Nov 03 20:58:14.010621 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:14.010591    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 20:58:14.626228 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:14.626199    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:58:14.626427 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:14.626407    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy" containerID="cri-o://106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5" gracePeriod=30
Nov 03 20:58:19.908612 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.908585    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:58:19.968594 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968568    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.968725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968637    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.968725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968660    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.968725 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968706    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.968902 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968749    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.968902 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968799    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.969012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968935    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.969012 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.968991    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:19.969156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969013    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.969156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969078    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\") pod \"93136739-ecc0-4939-a91b-fab66a741e7f\" (UID: \"93136739-ecc0-4939-a91b-fab66a741e7f\") "
Nov 03 20:58:19.969156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969103    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:19.969156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969104    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:19.969336 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969221    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:19.969404 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969355    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:19.969766 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969742    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:19.969860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969770    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:19.969860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969786    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93136739-ecc0-4939-a91b-fab66a741e7f-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:19.969860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969801    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:19.969860 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.969815    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:19.970945 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.970919    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:19.971109 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.971090    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:19.971169 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.971153    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:19.971265 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:19.971245    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x" pod "93136739-ecc0-4939-a91b-fab66a741e7f" (UID: "93136739-ecc0-4939-a91b-fab66a741e7f"). InnerVolumeSpecName "kube-api-access-9q68x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:20.070910 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.070881    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93136739-ecc0-4939-a91b-fab66a741e7f-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.071068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.070912    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.071068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.070927    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93136739-ecc0-4939-a91b-fab66a741e7f-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.071068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.070939    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q68x\" (UniqueName: \"kubernetes.io/projected/93136739-ecc0-4939-a91b-fab66a741e7f-kube-api-access-9q68x\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.132469 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.132429    4274 generic.go:356] "Generic (PLEG): container finished" podID="93136739-ecc0-4939-a91b-fab66a741e7f" containerID="106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5" exitCode=0
Nov 03 20:58:20.132641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.132486    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"
Nov 03 20:58:20.132641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.132506    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerDied","Data":"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"}
Nov 03 20:58:20.132641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.132583    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl" event={"ID":"93136739-ecc0-4939-a91b-fab66a741e7f","Type":"ContainerDied","Data":"223d7f711dd6fc5f01c2ca05c62b532e96d99b4629394e354436997906d310f4"}
Nov 03 20:58:20.132641 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.132604    4274 scope.go:122] "RemoveContainer" containerID="106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"
Nov 03 20:58:20.145483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.145466    4274 scope.go:122] "RemoveContainer" containerID="fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"
Nov 03 20:58:20.154963 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.154946    4274 scope.go:122] "RemoveContainer" containerID="106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"
Nov 03 20:58:20.155226 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:20.155207    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5\": container with ID starting with 106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5 not found: ID does not exist" containerID="106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"
Nov 03 20:58:20.155279 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.155235    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5"} err="failed to get container status \"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5\": rpc error: code = NotFound desc = could not find container \"106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5\": container with ID starting with 106e38ed14368752b6442e775685f4e2883bb3d03c20a8c63e29bcf8977547c5 not found: ID does not exist"
Nov 03 20:58:20.155279 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.155251    4274 scope.go:122] "RemoveContainer" containerID="fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"
Nov 03 20:58:20.155490 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:20.155475    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64\": container with ID starting with fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64 not found: ID does not exist" containerID="fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"
Nov 03 20:58:20.155529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.155493    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64"} err="failed to get container status \"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64\": rpc error: code = NotFound desc = could not find container \"fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64\": container with ID starting with fcd0875c7efa69e75c72e25a59a4a15ab37a00569da50047bc68ad2bd11d9f64 not found: ID does not exist"
Nov 03 20:58:20.287505 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.287477    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:58:20.294369 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:20.294340    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d9a2c67d-openshift-default-5d549998b5-q5nwl"]
Nov 03 20:58:21.214239 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:21.214208    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" path="/var/lib/kubelet/pods/93136739-ecc0-4939-a91b-fab66a741e7f/volumes"
Nov 03 20:58:22.883072 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.882994    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.883880    4274 cpu_manager.go:414] "containerMap: removing container" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.883905    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.883938    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f9c1357d-eb4a-4045-a942-4edff908f0a1" containerName="maas-api"
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.883945    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c1357d-eb4a-4045-a942-4edff908f0a1" containerName="maas-api"
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.884136    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:22.884332 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.884148    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c1357d-eb4a-4045-a942-4edff908f0a1" containerName="maas-api"
Nov 03 20:58:22.891249 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.891224    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:22.954083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.940700    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e68f8112-serving-cert\""
Nov 03 20:58:22.954083 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:22.947717    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 20:58:23.022211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.022177    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.022402 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.022279    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.022470 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.022449    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.022531 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.022479    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgbw\" (UniqueName: \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.124605 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.124569    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.124793 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.124611    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgbw\" (UniqueName: \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.124849 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.124802    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.124910 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.124893    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.124963 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:23.124942    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-serving-cert: secret "maas-api-e2e-mt-e68f8112-serving-cert" not found
Nov 03 20:58:23.125065 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:23.125024    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls podName:14d269e6-cdb0-4609-9c74-48c461f40bcc nodeName:}" failed. No retries permitted until 2026-11-03 20:58:23.625001811 +0000 UTC m=+1728.551572047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls") pod "maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc") : secret "maas-api-e2e-mt-e68f8112-serving-cert" not found
Nov 03 20:58:23.125377 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.125358    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.127719 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.127696    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.184532 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.184459    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgbw\" (UniqueName: \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.632678 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:23.632640    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:23.632866 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:23.632817    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-serving-cert: secret "maas-api-e2e-mt-e68f8112-serving-cert" not found
Nov 03 20:58:23.632920 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:23.632900    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls podName:14d269e6-cdb0-4609-9c74-48c461f40bcc nodeName:}" failed. No retries permitted until 2026-11-03 20:58:24.632882017 +0000 UTC m=+1729.559452236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls") pod "maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc") : secret "maas-api-e2e-mt-e68f8112-serving-cert" not found
Nov 03 20:58:24.647851 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:24.647817    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:24.650248 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:24.650218    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-68s8b\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:24.735715 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:24.735690    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:25.115453 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:25.115392    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 20:58:25.115453 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:58:25.115442    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d269e6_cdb0_4609_9c74_48c461f40bcc.slice/crio-1e9b8f93f78818f6d63a318ed5af47e515357475215a02e74a2a1621212dc476 WatchSource:0}: Error finding container 1e9b8f93f78818f6d63a318ed5af47e515357475215a02e74a2a1621212dc476: Status 404 returned error can't find the container with id 1e9b8f93f78818f6d63a318ed5af47e515357475215a02e74a2a1621212dc476
Nov 03 20:58:25.116950 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:25.116933    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:58:25.177740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:25.177715    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" event={"ID":"14d269e6-cdb0-4609-9c74-48c461f40bcc","Type":"ContainerStarted","Data":"1e9b8f93f78818f6d63a318ed5af47e515357475215a02e74a2a1621212dc476"}
Nov 03 20:58:26.185840 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:26.185805    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" event={"ID":"14d269e6-cdb0-4609-9c74-48c461f40bcc","Type":"ContainerStarted","Data":"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"}
Nov 03 20:58:26.186255 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:26.185883    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:27.366457 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:27.366419    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:58:32.196865 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:32.196836    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 20:58:32.307348 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:32.307302    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" podStartSLOduration=9.428264297 podStartE2EDuration="10.307288111s" totalImagesPullingTime="879.023814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:26.32450299 +0000 UTC m=+1731.251073229" watchObservedRunningTime="2026-11-03 20:58:32.307288111 +0000 UTC m=+1737.233858348"
Nov 03 20:58:32.691439 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:32.691405    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:58:32.691717 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:32.691621    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" podUID="690d8beb-ba5e-49d3-8929-c47aa32def2e" containerName="maas-api" containerID="cri-o://b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca" gracePeriod=30
Nov 03 20:58:32.996529 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:32.996503    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:58:33.172561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.171676    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") pod \"690d8beb-ba5e-49d3-8929-c47aa32def2e\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") "
Nov 03 20:58:33.172561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.171744    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\") pod \"690d8beb-ba5e-49d3-8929-c47aa32def2e\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") "
Nov 03 20:58:33.172561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.171822    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\") pod \"690d8beb-ba5e-49d3-8929-c47aa32def2e\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") "
Nov 03 20:58:33.172561 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.171953    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\" (UniqueName: \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\") pod \"690d8beb-ba5e-49d3-8929-c47aa32def2e\" (UID: \"690d8beb-ba5e-49d3-8929-c47aa32def2e\") "
Nov 03 20:58:33.172928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.172779    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca" pod "690d8beb-ba5e-49d3-8929-c47aa32def2e" (UID: "690d8beb-ba5e-49d3-8929-c47aa32def2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:33.173253 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.173218    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690d8beb-ba5e-49d3-8929-c47aa32def2e-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:33.177160 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.177106    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls" pod "690d8beb-ba5e-49d3-8929-c47aa32def2e" (UID: "690d8beb-ba5e-49d3-8929-c47aa32def2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:33.177434 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.177362    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s" pod "690d8beb-ba5e-49d3-8929-c47aa32def2e" (UID: "690d8beb-ba5e-49d3-8929-c47aa32def2e"). InnerVolumeSpecName "kube-api-access-vrc4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:33.177760 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.177735    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs" pod "690d8beb-ba5e-49d3-8929-c47aa32def2e" (UID: "690d8beb-ba5e-49d3-8929-c47aa32def2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:33.231767 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.231736    4274 generic.go:356] "Generic (PLEG): container finished" podID="690d8beb-ba5e-49d3-8929-c47aa32def2e" containerID="b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca" exitCode=0
Nov 03 20:58:33.232148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.231784    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"
Nov 03 20:58:33.232148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.231816    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" event={"ID":"690d8beb-ba5e-49d3-8929-c47aa32def2e","Type":"ContainerDied","Data":"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"}
Nov 03 20:58:33.232148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.231868    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd" event={"ID":"690d8beb-ba5e-49d3-8929-c47aa32def2e","Type":"ContainerDied","Data":"0ec672149d9d242bb659decec6bd24f9bdff2fe3131b7126bd3684013cab750a"}
Nov 03 20:58:33.232148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.231889    4274 scope.go:122] "RemoveContainer" containerID="b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"
Nov 03 20:58:33.242105 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.242088    4274 scope.go:122] "RemoveContainer" containerID="b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"
Nov 03 20:58:33.242382 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:33.242361    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca\": container with ID starting with b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca not found: ID does not exist" containerID="b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"
Nov 03 20:58:33.242454 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.242392    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca"} err="failed to get container status \"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca\": rpc error: code = NotFound desc = could not find container \"b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca\": container with ID starting with b9373782e3dc831e9d31fb32626cfaad515c3b9d7a50267e2bdfa55ed3905bca not found: ID does not exist"
Nov 03 20:58:33.274648 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.274621    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc4s\" (UniqueName: \"kubernetes.io/projected/690d8beb-ba5e-49d3-8929-c47aa32def2e-kube-api-access-vrc4s\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:33.274648 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.274651    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:33.274794 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.274661    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690d8beb-ba5e-49d3-8929-c47aa32def2e-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:33.391981 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.391949    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:58:33.396419 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.396389    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df4313ad-c67c976c9-dvcvd"]
Nov 03 20:58:33.846058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.845677    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:58:33.846058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:33.845936    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" podUID="d3c95f2a-29da-40c4-8d06-e089689134dd" containerName="maas-api" containerID="cri-o://0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05" gracePeriod=30
Nov 03 20:58:34.202796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.202765    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:58:34.239071 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.239014    4274 generic.go:356] "Generic (PLEG): container finished" podID="d3c95f2a-29da-40c4-8d06-e089689134dd" containerID="0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05" exitCode=0
Nov 03 20:58:34.239465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.239071    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" event={"ID":"d3c95f2a-29da-40c4-8d06-e089689134dd","Type":"ContainerDied","Data":"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"}
Nov 03 20:58:34.239465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.239086    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"
Nov 03 20:58:34.239465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.239128    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n" event={"ID":"d3c95f2a-29da-40c4-8d06-e089689134dd","Type":"ContainerDied","Data":"4a50fe492366998b3e80e818fe794f7f98b52a0d6965dc1a8109e22422bfc877"}
Nov 03 20:58:34.239465 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.239157    4274 scope.go:122] "RemoveContainer" containerID="0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"
Nov 03 20:58:34.251271 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.251252    4274 scope.go:122] "RemoveContainer" containerID="0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"
Nov 03 20:58:34.251565 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:34.251547    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05\": container with ID starting with 0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05 not found: ID does not exist" containerID="0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"
Nov 03 20:58:34.251627 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.251576    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05"} err="failed to get container status \"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05\": rpc error: code = NotFound desc = could not find container \"0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05\": container with ID starting with 0798c4f0b1b54910423793a9618a3b404fd2d70e7fb9055f28e64a7f627d0c05 not found: ID does not exist"
Nov 03 20:58:34.295711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.295671    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\" (UniqueName: \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\") pod \"d3c95f2a-29da-40c4-8d06-e089689134dd\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") "
Nov 03 20:58:34.295890 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.295775    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\") pod \"d3c95f2a-29da-40c4-8d06-e089689134dd\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") "
Nov 03 20:58:34.295890 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.295875    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") pod \"d3c95f2a-29da-40c4-8d06-e089689134dd\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") "
Nov 03 20:58:34.296168 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.296147    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\") pod \"d3c95f2a-29da-40c4-8d06-e089689134dd\" (UID: \"d3c95f2a-29da-40c4-8d06-e089689134dd\") "
Nov 03 20:58:34.296274 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.296149    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca" pod "d3c95f2a-29da-40c4-8d06-e089689134dd" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:34.297796 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.297769    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c95f2a-29da-40c4-8d06-e089689134dd-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:34.298477 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.298450    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls" pod "d3c95f2a-29da-40c4-8d06-e089689134dd" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:34.298590 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.298536    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658" pod "d3c95f2a-29da-40c4-8d06-e089689134dd" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd"). InnerVolumeSpecName "kube-api-access-t7658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:34.298984 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.298962    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs" pod "d3c95f2a-29da-40c4-8d06-e089689134dd" (UID: "d3c95f2a-29da-40c4-8d06-e089689134dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:34.398763 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.398729    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:34.398763 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.398757    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7658\" (UniqueName: \"kubernetes.io/projected/d3c95f2a-29da-40c4-8d06-e089689134dd-kube-api-access-t7658\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:34.398763 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.398767    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c95f2a-29da-40c4-8d06-e089689134dd-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:34.654636 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.654599    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 20:58:34.655361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655334    4274 cpu_manager.go:414] "containerMap: removing container" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:34.655361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655361    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:34.655571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655402    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d3c95f2a-29da-40c4-8d06-e089689134dd" containerName="maas-api"
Nov 03 20:58:34.655571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655421    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c95f2a-29da-40c4-8d06-e089689134dd" containerName="maas-api"
Nov 03 20:58:34.655571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655462    4274 cpu_manager.go:414] "containerMap: removing container" podUID="690d8beb-ba5e-49d3-8929-c47aa32def2e" containerName="maas-api"
Nov 03 20:58:34.655571 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655469    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690d8beb-ba5e-49d3-8929-c47aa32def2e" containerName="maas-api"
Nov 03 20:58:34.655739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655627    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="93136739-ecc0-4939-a91b-fab66a741e7f" containerName="istio-proxy"
Nov 03 20:58:34.655739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655641    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c95f2a-29da-40c4-8d06-e089689134dd" containerName="maas-api"
Nov 03 20:58:34.655739 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.655662    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="690d8beb-ba5e-49d3-8929-c47aa32def2e" containerName="maas-api"
Nov 03 20:58:34.661904 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.661883    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.669870 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.669842    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:58:34.677374 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.677350    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-299a741b-54b9759995-mdh7n"]
Nov 03 20:58:34.711399 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.711370    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ba7e83\""
Nov 03 20:58:34.712277 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.712215    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ba7e83-dockercfg-xfsqm\""
Nov 03 20:58:34.717022 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.716993    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 20:58:34.787055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.777085    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 20:58:34.787055 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.786399    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:34.804973 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.804806    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lxm\" (UniqueName: \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.805156 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.805131    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.879155 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.873229    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 20:58:34.915581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.913027    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:34.915581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.913239    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scxt\" (UniqueName: \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:34.915581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.913307    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.915581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.913440    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lxm\" (UniqueName: \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.915581 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.914734    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:34.989251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:34.989218    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lxm\" (UniqueName: \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\") pod \"payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:35.019058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.015216    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:35.019058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.015346    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scxt\" (UniqueName: \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:35.019058 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.016005    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:35.073761 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.073722    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scxt\" (UniqueName: \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\") pod \"payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") " pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:35.121132 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.121097    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:35.236263 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.236231    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690d8beb-ba5e-49d3-8929-c47aa32def2e" path="/var/lib/kubelet/pods/690d8beb-ba5e-49d3-8929-c47aa32def2e/volumes"
Nov 03 20:58:35.236867 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.236687    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c95f2a-29da-40c4-8d06-e089689134dd" path="/var/lib/kubelet/pods/d3c95f2a-29da-40c4-8d06-e089689134dd/volumes"
Nov 03 20:58:35.283695 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.281821    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:35.513321 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.513295    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 20:58:35.688148 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:35.688124    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 20:58:35.689721 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:58:35.689686    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f00c265_f625_4797_834f_cacad9b68ab0.slice/crio-bcb438363c07f4e33ebb20c91aa08e58a85ac1cd8c519b2fb1513319990d3339 WatchSource:0}: Error finding container bcb438363c07f4e33ebb20c91aa08e58a85ac1cd8c519b2fb1513319990d3339: Status 404 returned error can't find the container with id bcb438363c07f4e33ebb20c91aa08e58a85ac1cd8c519b2fb1513319990d3339
Nov 03 20:58:36.269809 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.269773    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" event={"ID":"bae534ad-9d21-4d8a-88be-b66d56b08eef","Type":"ContainerStarted","Data":"a9fbaf8f60f7b30b7a5bf68a1082e0c26eb4dd98c1571b94184a21265901fb05"}
Nov 03 20:58:36.270000 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.269834    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" event={"ID":"bae534ad-9d21-4d8a-88be-b66d56b08eef","Type":"ContainerStarted","Data":"ee6522b46abf6a5e2e54871ebe190ffc75d8ab46b2450a4653829d1dd9d78c45"}
Nov 03 20:58:36.270000 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.269878    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:36.271713 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.271690    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" event={"ID":"4f00c265-f625-4797-834f-cacad9b68ab0","Type":"ContainerStarted","Data":"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"}
Nov 03 20:58:36.271859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.271727    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" event={"ID":"4f00c265-f625-4797-834f-cacad9b68ab0","Type":"ContainerStarted","Data":"bcb438363c07f4e33ebb20c91aa08e58a85ac1cd8c519b2fb1513319990d3339"}
Nov 03 20:58:36.271859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.271779    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:36.395414 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.395359    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" podStartSLOduration=2.39534246 podStartE2EDuration="2.39534246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:36.392297094 +0000 UTC m=+1741.318867330" watchObservedRunningTime="2026-11-03 20:58:36.39534246 +0000 UTC m=+1741.321912696"
Nov 03 20:58:36.505115 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:36.505050    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" podStartSLOduration=2.505018634 podStartE2EDuration="2.505018634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:36.500163424 +0000 UTC m=+1741.426733672" watchObservedRunningTime="2026-11-03 20:58:36.505018634 +0000 UTC m=+1741.431588871"
Nov 03 20:58:37.991538 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:37.991499    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 20:58:38.005491 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.005459    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.074068 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.068988    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2e3ea2-openshift-default-dockercfg-hc8bk\""
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108056    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108136    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108201    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108230    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108263    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108358    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcf2\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.123089 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.108399    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.188196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.174507    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.188196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.174582    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.211226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.208055    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280112    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280180    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280261    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280306    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280366    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280400    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280427    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280500    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcf2\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.281056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.280534    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.293173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.281406    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.293173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.282097    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.293173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.284330    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.293173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.284559    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.293173 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.284790    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.300064 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.298337    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.300064 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.298809    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.338840 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.338810    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.343888 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.343852    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcf2\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\") pod \"e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") " pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:38.676112 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:38.668719    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:39.248842 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:39.248814    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 20:58:39.249343 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:58:39.248852    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24e01f2_dffc_4769_aad9_43dde43e2094.slice/crio-e19190039c8861c2efb692187124a5ad78d5ae2f311c69c3505109074f8ac9e9 WatchSource:0}: Error finding container e19190039c8861c2efb692187124a5ad78d5ae2f311c69c3505109074f8ac9e9: Status 404 returned error can't find the container with id e19190039c8861c2efb692187124a5ad78d5ae2f311c69c3505109074f8ac9e9
Nov 03 20:58:39.252163 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:39.252129    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:39.252256 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:39.252195    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:39.252256 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:39.252224    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:39.320225 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:39.320194    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerStarted","Data":"e19190039c8861c2efb692187124a5ad78d5ae2f311c69c3505109074f8ac9e9"}
Nov 03 20:58:40.327994 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:40.327961    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerStarted","Data":"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"}
Nov 03 20:58:40.443791 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:40.443731    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podStartSLOduration=3.443713327 podStartE2EDuration="3.443713327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:40.441066461 +0000 UTC m=+1745.367636702" watchObservedRunningTime="2026-11-03 20:58:40.443713327 +0000 UTC m=+1745.370283633"
Nov 03 20:58:40.671984 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:40.671806    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:40.676102 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:40.676069    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:41.334462 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:41.334432    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:41.335509 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:41.335486    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 20:58:46.438599 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.438519    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 20:58:46.443069 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.443024    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.491318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.491289    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-5b12d6-serving-cert\""
Nov 03 20:58:46.499597 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.499568    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 20:58:46.536144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.536119    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2cv\" (UniqueName: \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.536266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.536163    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.536266 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.536248    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.536353 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.536287    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.637318 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.637285    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.637489 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.637351    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.637567 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.637555    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2cv\" (UniqueName: \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.637631 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.637615    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.637793 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:46.637774    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-serving-cert: secret "maas-api-e2e-shared-a-w6-5b12d6-serving-cert" not found
Nov 03 20:58:46.637871 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:46.637848    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls podName:4fa3748f-ee00-48c0-bfc2-29fec5e25727 nodeName:}" failed. No retries permitted until 2026-11-03 20:58:47.137828784 +0000 UTC m=+1752.064399004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls") pod "maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" (UID: "4fa3748f-ee00-48c0-bfc2-29fec5e25727") : secret "maas-api-e2e-shared-a-w6-5b12d6-serving-cert" not found
Nov 03 20:58:46.638184 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.638160    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.640164 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.640140    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:46.697946 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:46.697874    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2cv\" (UniqueName: \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:47.156293 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.156258    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:47.158902 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.158878    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:47.299757 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.299724    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 20:58:47.302977 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.302951    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 20:58:47.359088 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.358091    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:47.744211 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:47.744182    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 20:58:47.744987 ip-10-0-128-118 kubenswrapper[4274]: W1103 20:58:47.744954    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa3748f_ee00_48c0_bfc2_29fec5e25727.slice/crio-4c4ef2193fe1bd3a37a1da51b02f62f57662732891a73341464f58fe7ae6753c WatchSource:0}: Error finding container 4c4ef2193fe1bd3a37a1da51b02f62f57662732891a73341464f58fe7ae6753c: Status 404 returned error can't find the container with id 4c4ef2193fe1bd3a37a1da51b02f62f57662732891a73341464f58fe7ae6753c
Nov 03 20:58:48.394023 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:48.393990    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" event={"ID":"4fa3748f-ee00-48c0-bfc2-29fec5e25727","Type":"ContainerStarted","Data":"4c4ef2193fe1bd3a37a1da51b02f62f57662732891a73341464f58fe7ae6753c"}
Nov 03 20:58:49.403056 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:49.403007    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" event={"ID":"4fa3748f-ee00-48c0-bfc2-29fec5e25727","Type":"ContainerStarted","Data":"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"}
Nov 03 20:58:49.403430 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:49.403180    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:49.513638 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:49.513589    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" podStartSLOduration=2.651868416 podStartE2EDuration="3.513575896s" totalImagesPullingTime="861.70748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:49.512437171 +0000 UTC m=+1754.439007409" watchObservedRunningTime="2026-11-03 20:58:49.513575896 +0000 UTC m=+1754.440146132"
Nov 03 20:58:55.413759 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:58:55.413732    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 20:58:57.367434 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:58:57.367398    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:59:27.367168 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:59:27.367136    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 20:59:27.974986 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:27.974952    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:59:27.975196 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:27.975158    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy" containerID="cri-o://4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524" gracePeriod=30
Nov 03 20:59:27.978436 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:27.978409    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:59:27.978613 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:27.978593    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy" containerID="cri-o://621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6" gracePeriod=30
Nov 03 20:59:33.271514 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.271467    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:59:33.308488 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.308463    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:59:33.354152 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354126    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.354276 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354167    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.354276 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354210    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.354361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354290    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.354361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354331    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.354511 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354489    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.354704 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354625    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354670    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354674    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354687    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354773    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354786    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.354830 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354822    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.355214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354861    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354936    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354964    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.355214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.354999    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355214 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355116    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355213    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355258    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355302    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355307    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355344    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\") pod \"013ba7f6-a51d-430b-93c4-ec241723c414\" (UID: \"013ba7f6-a51d-430b-93c4-ec241723c414\") "
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355388    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\") pod \"2686e8e2-1901-46fd-a36b-df2ada76ed59\" (UID: \"2686e8e2-1901-46fd-a36b-df2ada76ed59\") "
Nov 03 20:59:33.355483 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355474    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.355856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355485    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.355856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355660    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.355856 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.355808    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356142    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356170    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356185    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356199    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356212    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356226    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2686e8e2-1901-46fd-a36b-df2ada76ed59-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356241    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013ba7f6-a51d-430b-93c4-ec241723c414-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356256    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356270    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356306 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356284    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.356949 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.356910    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.357134 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.357108    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:33.357221 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.357194    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:33.358236 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.358206    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "kube-api-access-978mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:33.358350 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.358234    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:33.358420 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.358367    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:33.358642 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.358622    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy" pod "2686e8e2-1901-46fd-a36b-df2ada76ed59" (UID: "2686e8e2-1901-46fd-a36b-df2ada76ed59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:33.359146 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.359126    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq" pod "013ba7f6-a51d-430b-93c4-ec241723c414" (UID: "013ba7f6-a51d-430b-93c4-ec241723c414"). InnerVolumeSpecName "kube-api-access-jk9wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:33.457556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457516    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9wq\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-kube-api-access-jk9wq\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457556 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457558    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013ba7f6-a51d-430b-93c4-ec241723c414-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457574    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978mx\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-kube-api-access-978mx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457591    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013ba7f6-a51d-430b-93c4-ec241723c414-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457605    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457619    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457633    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2686e8e2-1901-46fd-a36b-df2ada76ed59-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.457753 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.457645    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013ba7f6-a51d-430b-93c4-ec241723c414-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:33.721429 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.721385    4274 generic.go:356] "Generic (PLEG): container finished" podID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerID="4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524" exitCode=0
Nov 03 20:59:33.721608 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.721466    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"
Nov 03 20:59:33.721608 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.721512    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" event={"ID":"2686e8e2-1901-46fd-a36b-df2ada76ed59","Type":"ContainerDied","Data":"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"}
Nov 03 20:59:33.721703 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.721605    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx" event={"ID":"2686e8e2-1901-46fd-a36b-df2ada76ed59","Type":"ContainerDied","Data":"d4b716d0a2667ef9767671464c6894bff1fbbd19e354aab31033c82b58e6f234"}
Nov 03 20:59:33.721703 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.721640    4274 scope.go:122] "RemoveContainer" containerID="4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"
Nov 03 20:59:33.730798 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.730764    4274 generic.go:356] "Generic (PLEG): container finished" podID="013ba7f6-a51d-430b-93c4-ec241723c414" containerID="621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6" exitCode=0
Nov 03 20:59:33.730928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.730858    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"
Nov 03 20:59:33.730928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.730856    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" event={"ID":"013ba7f6-a51d-430b-93c4-ec241723c414","Type":"ContainerDied","Data":"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"}
Nov 03 20:59:33.730928 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.730913    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462" event={"ID":"013ba7f6-a51d-430b-93c4-ec241723c414","Type":"ContainerDied","Data":"5a92d3e039a6af4bc74180142905c7d01a0f11b911cffe556e6e62fc4c8e7e6c"}
Nov 03 20:59:33.743136 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.743113    4274 scope.go:122] "RemoveContainer" containerID="4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"
Nov 03 20:59:33.743496 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:59:33.743472    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524\": container with ID starting with 4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524 not found: ID does not exist" containerID="4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"
Nov 03 20:59:33.743550 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.743508    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524"} err="failed to get container status \"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524\": rpc error: code = NotFound desc = could not find container \"4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524\": container with ID starting with 4cbd02c20461d89a74210ee7b85c0d4805d9d217e579b62ab0f43b2252e8c524 not found: ID does not exist"
Nov 03 20:59:33.743550 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.743532    4274 scope.go:122] "RemoveContainer" containerID="621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"
Nov 03 20:59:33.752251 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.752234    4274 scope.go:122] "RemoveContainer" containerID="621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"
Nov 03 20:59:33.752512 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:59:33.752496    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6\": container with ID starting with 621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6 not found: ID does not exist" containerID="621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"
Nov 03 20:59:33.752558 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.752517    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6"} err="failed to get container status \"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6\": rpc error: code = NotFound desc = could not find container \"621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6\": container with ID starting with 621d1d30303489ca7f66bdf5082188de774bd970a77cace9867dadebd07b1ec6 not found: ID does not exist"
Nov 03 20:59:33.879021 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.878980    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:59:33.885681 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.885645    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4tmpx"]
Nov 03 20:59:33.990740 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.990658    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:59:33.997828 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:33.997800    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gm462"]
Nov 03 20:59:35.215888 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:35.215851    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" path="/var/lib/kubelet/pods/013ba7f6-a51d-430b-93c4-ec241723c414/volumes"
Nov 03 20:59:35.216412 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:35.216396    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" path="/var/lib/kubelet/pods/2686e8e2-1901-46fd-a36b-df2ada76ed59/volumes"
Nov 03 20:59:42.964242 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:42.964207    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:59:42.964722 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:42.964456    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy" containerID="cri-o://00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf" gracePeriod=30
Nov 03 20:59:48.227859 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.227803    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:59:48.313512 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313481    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313676 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313537    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313676 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313595    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313676 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313646    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313692    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313735    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313797    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.313845 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313843    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.314076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313878    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\") pod \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\" (UID: \"5147d69b-fa33-4ae8-a2ed-b9a16b66386b\") "
Nov 03 20:59:48.314076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313889    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.314076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313974    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.314076 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.313984    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.314263 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.314097    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.314263 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.314214    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:48.315014 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.314988    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.315162 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.315139    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.315226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.315161    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.315226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.315178    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.315226 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.315193    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.316144 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.316117    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:48.316242 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.316166    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:48.316361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.316341    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.316361 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.316350    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx" pod "5147d69b-fa33-4ae8-a2ed-b9a16b66386b" (UID: "5147d69b-fa33-4ae8-a2ed-b9a16b66386b"). InnerVolumeSpecName "kube-api-access-zvlpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:48.417153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.417061    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.417153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.417089    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.417153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.417098    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.417153 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.417107    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvlpx\" (UniqueName: \"kubernetes.io/projected/5147d69b-fa33-4ae8-a2ed-b9a16b66386b-kube-api-access-zvlpx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.838519 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.838479    4274 generic.go:356] "Generic (PLEG): container finished" podID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerID="00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf" exitCode=0
Nov 03 20:59:48.838691 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.838534    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"
Nov 03 20:59:48.838691 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.838568    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" event={"ID":"5147d69b-fa33-4ae8-a2ed-b9a16b66386b","Type":"ContainerDied","Data":"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"}
Nov 03 20:59:48.838691 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.838638    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd" event={"ID":"5147d69b-fa33-4ae8-a2ed-b9a16b66386b","Type":"ContainerDied","Data":"4010b14a4ed909e74852b940d826a739d763206645bc6591d9aba7480483bd6c"}
Nov 03 20:59:48.838691 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.838662    4274 scope.go:122] "RemoveContainer" containerID="00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"
Nov 03 20:59:48.849911 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.849894    4274 scope.go:122] "RemoveContainer" containerID="00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"
Nov 03 20:59:48.850293 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:59:48.850271    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf\": container with ID starting with 00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf not found: ID does not exist" containerID="00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"
Nov 03 20:59:48.850441 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.850302    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf"} err="failed to get container status \"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf\": rpc error: code = NotFound desc = could not find container \"00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf\": container with ID starting with 00858bf4d3bbaf503024b432b3f033d2065081de687c5cecf21e262cd94302bf not found: ID does not exist"
Nov 03 20:59:48.993653 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:48.993594    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:59:49.039711 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:49.039674    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-hckdd"]
Nov 03 20:59:49.215398 ip-10-0-128-118 kubenswrapper[4274]: I1103 20:59:49.215307    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" path="/var/lib/kubelet/pods/5147d69b-fa33-4ae8-a2ed-b9a16b66386b/volumes"
Nov 03 20:59:57.367966 ip-10-0-128-118 kubenswrapper[4274]: E1103 20:59:57.367912    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:00:00.233835 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.233804    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:00.234476 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234459    4274 cpu_manager.go:414] "containerMap: removing container" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy"
Nov 03 21:00:00.234540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234477    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy"
Nov 03 21:00:00.234540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234503    4274 cpu_manager.go:414] "containerMap: removing container" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy"
Nov 03 21:00:00.234540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234509    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy"
Nov 03 21:00:00.234540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234518    4274 cpu_manager.go:414] "containerMap: removing container" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy"
Nov 03 21:00:00.234540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234523    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy"
Nov 03 21:00:00.234704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234615    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5147d69b-fa33-4ae8-a2ed-b9a16b66386b" containerName="istio-proxy"
Nov 03 21:00:00.234704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234631    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2686e8e2-1901-46fd-a36b-df2ada76ed59" containerName="istio-proxy"
Nov 03 21:00:00.234704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.234641    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="013ba7f6-a51d-430b-93c4-ec241723c414" containerName="istio-proxy"
Nov 03 21:00:00.237685 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.237668    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.258967 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.258944    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:00.263091 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.263072    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.321202 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.321167    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:00.332980 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.332952    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:00:00.367313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.367280    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:00.388391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.388356    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.388564 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.388549    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djnv\" (UniqueName: \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.388628 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.388586    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.389030 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.388996    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txk2r\" (UniqueName: \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.490865 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.490781    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djnv\" (UniqueName: \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.490865 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.490825    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.491129 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.490919    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk2r\" (UniqueName: \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.491129 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.491009    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.491526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.491505    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.491677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.491659    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.552618 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.552583    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djnv\" (UniqueName: \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.552735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.552618    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk2r\" (UniqueName: \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.583349 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.583283    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:00.852142 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.852113    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:00.958479 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:00.958434    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:00.959078 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:00:00.959047    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85639f6e_1f74_4502_880c_55da7cf7fa4b.slice/crio-0f744074d4836ff5d6c42594b4c96fca5f75d5685a21d0eb4c78b3d1ecef5ffa WatchSource:0}: Error finding container 0f744074d4836ff5d6c42594b4c96fca5f75d5685a21d0eb4c78b3d1ecef5ffa: Status 404 returned error can't find the container with id 0f744074d4836ff5d6c42594b4c96fca5f75d5685a21d0eb4c78b3d1ecef5ffa
Nov 03 21:00:01.221153 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.221119    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:01.222461 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:00:01.222435    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd176d16b_3f6e_4c8d_998a_ad2959b5c099.slice/crio-20832f527e652580d71be08181dadf8303bbe7adc9eb49a7135e9d0812998a2d WatchSource:0}: Error finding container 20832f527e652580d71be08181dadf8303bbe7adc9eb49a7135e9d0812998a2d: Status 404 returned error can't find the container with id 20832f527e652580d71be08181dadf8303bbe7adc9eb49a7135e9d0812998a2d
Nov 03 21:00:01.922049 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.921990    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerStarted","Data":"4ce52b2dd39b9063714bec5aafdc8dc54afb33e7503661d3379a7fb6e0d6ff40"}
Nov 03 21:00:01.922484 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.922080    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerStarted","Data":"20832f527e652580d71be08181dadf8303bbe7adc9eb49a7135e9d0812998a2d"}
Nov 03 21:00:01.923605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.923579    4274 generic.go:356] "Generic (PLEG): container finished" podID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerID="1f0f423885def6069cc4b6f64e49a771be5a7d9632f98b6889628176b7e02b43" exitCode=60
Nov 03 21:00:01.923760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.923658    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerDied","Data":"1f0f423885def6069cc4b6f64e49a771be5a7d9632f98b6889628176b7e02b43"}
Nov 03 21:00:01.923760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.923699    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerStarted","Data":"0f744074d4836ff5d6c42594b4c96fca5f75d5685a21d0eb4c78b3d1ecef5ffa"}
Nov 03 21:00:01.923952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:01.923934    4274 scope.go:122] "RemoveContainer" containerID="1f0f423885def6069cc4b6f64e49a771be5a7d9632f98b6889628176b7e02b43"
Nov 03 21:00:02.033366 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:02.033311    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" podStartSLOduration=2.033297073 podStartE2EDuration="2.033297073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.029729673 +0000 UTC m=+1826.956299912" watchObservedRunningTime="2026-11-03 21:00:02.033297073 +0000 UTC m=+1826.959867309"
Nov 03 21:00:02.930949 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:02.930915    4274 generic.go:356] "Generic (PLEG): container finished" podID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerID="4ce52b2dd39b9063714bec5aafdc8dc54afb33e7503661d3379a7fb6e0d6ff40" exitCode=60
Nov 03 21:00:02.931366 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:02.931002    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerDied","Data":"4ce52b2dd39b9063714bec5aafdc8dc54afb33e7503661d3379a7fb6e0d6ff40"}
Nov 03 21:00:02.931366 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:02.931279    4274 scope.go:122] "RemoveContainer" containerID="4ce52b2dd39b9063714bec5aafdc8dc54afb33e7503661d3379a7fb6e0d6ff40"
Nov 03 21:00:02.933022 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:02.933002    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerStarted","Data":"f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"}
Nov 03 21:00:03.147863 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.147817    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" podStartSLOduration=3.147802075 podStartE2EDuration="3.147802075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.147475081 +0000 UTC m=+1828.074045321" watchObservedRunningTime="2026-11-03 21:00:03.147802075 +0000 UTC m=+1828.074372312"
Nov 03 21:00:03.941838 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.941800    4274 generic.go:356] "Generic (PLEG): container finished" podID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerID="243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471" exitCode=60
Nov 03 21:00:03.942281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.941847    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerDied","Data":"243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471"}
Nov 03 21:00:03.942281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.941904    4274 scope.go:122] "RemoveContainer" containerID="4ce52b2dd39b9063714bec5aafdc8dc54afb33e7503661d3379a7fb6e0d6ff40"
Nov 03 21:00:03.942281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.942202    4274 scope.go:122] "RemoveContainer" containerID="243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471"
Nov 03 21:00:03.942561 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:03.942538    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q_odh-ai-gateway-infra(d176d16b-3f6e-4c8d-998a-ad2959b5c099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099"
Nov 03 21:00:03.943894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.943869    4274 generic.go:356] "Generic (PLEG): container finished" podID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerID="f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320" exitCode=60
Nov 03 21:00:03.944001 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.943937    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerDied","Data":"f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"}
Nov 03 21:00:03.944227 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.944212    4274 scope.go:122] "RemoveContainer" containerID="f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"
Nov 03 21:00:03.944469 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:03.944451    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc_odh-ai-gateway-infra(85639f6e-1f74-4502-880c-55da7cf7fa4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b"
Nov 03 21:00:03.956648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:03.956625    4274 scope.go:122] "RemoveContainer" containerID="1f0f423885def6069cc4b6f64e49a771be5a7d9632f98b6889628176b7e02b43"
Nov 03 21:00:04.952307 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:04.952281    4274 scope.go:122] "RemoveContainer" containerID="f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"
Nov 03 21:00:04.952737 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:04.952555    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc_odh-ai-gateway-infra(85639f6e-1f74-4502-880c-55da7cf7fa4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b"
Nov 03 21:00:04.954015 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:04.953998    4274 scope.go:122] "RemoveContainer" containerID="243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471"
Nov 03 21:00:04.954238 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:04.954220    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q_odh-ai-gateway-infra(d176d16b-3f6e-4c8d-998a-ad2959b5c099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099"
Nov 03 21:00:15.210169 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.210092    4274 scope.go:122] "RemoveContainer" containerID="243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471"
Nov 03 21:00:15.482058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.481964    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:00:15.487448 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.487426    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.584116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.584082    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-9ce672-serving-cert\""
Nov 03 21:00:15.587764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.587731    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:00:15.613056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.613014    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.613200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.613150    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.613316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.613300    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.613379 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.613345    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxrp\" (UniqueName: \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.721561 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.721505    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.722815 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.722777    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.724350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.724249    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.724350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.724301    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxrp\" (UniqueName: \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.724350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.724252    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.724350 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:15.724337    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-serving-cert: secret "maas-api-e2e-shared-b-w5-9ce672-serving-cert" not found
Nov 03 21:00:15.724350 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:15.724398    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls podName:b0215b97-42ea-404d-9afa-fc82cbbf0740 nodeName:}" failed. No retries permitted until 2026-11-03 21:00:16.224377822 +0000 UTC m=+1841.150948051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls") pod "maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" (UID: "b0215b97-42ea-404d-9afa-fc82cbbf0740") : secret "maas-api-e2e-shared-b-w5-9ce672-serving-cert" not found
Nov 03 21:00:15.738775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.728394    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:15.787904 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:15.787871    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxrp\" (UniqueName: \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:16.099018 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:16.098982    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerStarted","Data":"d661348f7e8b549aa3ba9499d2944e5ae16d0801219ef10f3853a6a77e828d11"}
Nov 03 21:00:16.230797 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:16.230757    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:16.233145 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:16.233126    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:16.401241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:16.401174    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:16.817787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:16.817759    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:00:16.820025 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:00:16.819995    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0215b97_42ea_404d_9afa_fc82cbbf0740.slice/crio-064b1a196cf7e4ea6534fa3236358a52dd0ea4beac0e3fb7cd345eef5e9ea07d WatchSource:0}: Error finding container 064b1a196cf7e4ea6534fa3236358a52dd0ea4beac0e3fb7cd345eef5e9ea07d: Status 404 returned error can't find the container with id 064b1a196cf7e4ea6534fa3236358a52dd0ea4beac0e3fb7cd345eef5e9ea07d
Nov 03 21:00:17.107159 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.107066    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" event={"ID":"b0215b97-42ea-404d-9afa-fc82cbbf0740","Type":"ContainerStarted","Data":"064b1a196cf7e4ea6534fa3236358a52dd0ea4beac0e3fb7cd345eef5e9ea07d"}
Nov 03 21:00:17.108871 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.108842    4274 generic.go:356] "Generic (PLEG): container finished" podID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerID="d661348f7e8b549aa3ba9499d2944e5ae16d0801219ef10f3853a6a77e828d11" exitCode=60
Nov 03 21:00:17.108991 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.108907    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerDied","Data":"d661348f7e8b549aa3ba9499d2944e5ae16d0801219ef10f3853a6a77e828d11"}
Nov 03 21:00:17.108991 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.108965    4274 scope.go:122] "RemoveContainer" containerID="243f850f3934f2c91640673ce64c612425f47004b6c9d58353701a49d3577471"
Nov 03 21:00:17.109332 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.109316    4274 scope.go:122] "RemoveContainer" containerID="d661348f7e8b549aa3ba9499d2944e5ae16d0801219ef10f3853a6a77e828d11"
Nov 03 21:00:17.109653 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:17.109635    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q_odh-ai-gateway-infra(d176d16b-3f6e-4c8d-998a-ad2959b5c099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099"
Nov 03 21:00:17.214722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.214697    4274 scope.go:122] "RemoveContainer" containerID="f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"
Nov 03 21:00:17.223326 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:17.223301    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:18.119975 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.119940    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" event={"ID":"b0215b97-42ea-404d-9afa-fc82cbbf0740","Type":"ContainerStarted","Data":"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"}
Nov 03 21:00:18.122554 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.120024    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:18.124687 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.124659    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerStarted","Data":"51934e1fb98244fe39c746ced2b3d06e61f93bb7353b1a67cd829b062b84079f"}
Nov 03 21:00:18.234745 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.234685    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" podStartSLOduration=2.316952477 podStartE2EDuration="3.23466545s" totalImagesPullingTime="917.712973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:18.229831393 +0000 UTC m=+1843.156401630" watchObservedRunningTime="2026-11-03 21:00:18.23466545 +0000 UTC m=+1843.161235682"
Nov 03 21:00:18.278425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.278402    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:18.365675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.364586    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\") pod \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") "
Nov 03 21:00:18.365675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.364791    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\" (UniqueName: \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\") pod \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\" (UID: \"d176d16b-3f6e-4c8d-998a-ad2959b5c099\") "
Nov 03 21:00:18.367060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.367000    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca" pod "d176d16b-3f6e-4c8d-998a-ad2959b5c099" (UID: "d176d16b-3f6e-4c8d-998a-ad2959b5c099"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:18.367869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.367841    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d176d16b-3f6e-4c8d-998a-ad2959b5c099-maas-api-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.370697 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.370634    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv" pod "d176d16b-3f6e-4c8d-998a-ad2959b5c099" (UID: "d176d16b-3f6e-4c8d-998a-ad2959b5c099"). InnerVolumeSpecName "kube-api-access-2djnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:18.469859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:18.469819    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2djnv\" (UniqueName: \"kubernetes.io/projected/d176d16b-3f6e-4c8d-998a-ad2959b5c099-kube-api-access-2djnv\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.133881 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.133850    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"
Nov 03 21:00:19.134398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.133886    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q" event={"ID":"d176d16b-3f6e-4c8d-998a-ad2959b5c099","Type":"ContainerDied","Data":"20832f527e652580d71be08181dadf8303bbe7adc9eb49a7135e9d0812998a2d"}
Nov 03 21:00:19.134398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.133958    4274 scope.go:122] "RemoveContainer" containerID="d661348f7e8b549aa3ba9499d2944e5ae16d0801219ef10f3853a6a77e828d11"
Nov 03 21:00:19.135966 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.135939    4274 generic.go:356] "Generic (PLEG): container finished" podID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerID="51934e1fb98244fe39c746ced2b3d06e61f93bb7353b1a67cd829b062b84079f" exitCode=60
Nov 03 21:00:19.136868 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.136843    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerDied","Data":"51934e1fb98244fe39c746ced2b3d06e61f93bb7353b1a67cd829b062b84079f"}
Nov 03 21:00:19.137238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.137220    4274 scope.go:122] "RemoveContainer" containerID="51934e1fb98244fe39c746ced2b3d06e61f93bb7353b1a67cd829b062b84079f"
Nov 03 21:00:19.137517 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:19.137496    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc_odh-ai-gateway-infra(85639f6e-1f74-4502-880c-55da7cf7fa4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b"
Nov 03 21:00:19.152649 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.152630    4274 scope.go:122] "RemoveContainer" containerID="f97192917ed4860bb6775efc2352645aa7b4096557e5c4b484a0740d1308a320"
Nov 03 21:00:19.355674 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.355643    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:19.402663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.402592    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:19.411647 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:19.411618    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895660-fqz2q"]
Nov 03 21:00:20.317964 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.317940    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:20.398232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.398175    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\" (UniqueName: \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\") pod \"85639f6e-1f74-4502-880c-55da7cf7fa4b\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") "
Nov 03 21:00:20.398232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.398215    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\") pod \"85639f6e-1f74-4502-880c-55da7cf7fa4b\" (UID: \"85639f6e-1f74-4502-880c-55da7cf7fa4b\") "
Nov 03 21:00:20.398541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.398517    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca" pod "85639f6e-1f74-4502-880c-55da7cf7fa4b" (UID: "85639f6e-1f74-4502-880c-55da7cf7fa4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:20.398806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.398792    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85639f6e-1f74-4502-880c-55da7cf7fa4b-maas-api-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.400222 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.400202    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r" pod "85639f6e-1f74-4502-880c-55da7cf7fa4b" (UID: "85639f6e-1f74-4502-880c-55da7cf7fa4b"). InnerVolumeSpecName "kube-api-access-txk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:20.500720 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:20.500686    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txk2r\" (UniqueName: \"kubernetes.io/projected/85639f6e-1f74-4502-880c-55da7cf7fa4b-kube-api-access-txk2r\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:21.154456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.154421    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"
Nov 03 21:00:21.154456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.154442    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc" event={"ID":"85639f6e-1f74-4502-880c-55da7cf7fa4b","Type":"ContainerDied","Data":"0f744074d4836ff5d6c42594b4c96fca5f75d5685a21d0eb4c78b3d1ecef5ffa"}
Nov 03 21:00:21.154720 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.154501    4274 scope.go:122] "RemoveContainer" containerID="51934e1fb98244fe39c746ced2b3d06e61f93bb7353b1a67cd829b062b84079f"
Nov 03 21:00:21.225645 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.225603    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" path="/var/lib/kubelet/pods/d176d16b-3f6e-4c8d-998a-ad2959b5c099/volumes"
Nov 03 21:00:21.331823 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.331791    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:21.339608 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:21.339582    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895660-2kthc"]
Nov 03 21:00:23.214860 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:23.214826    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" path="/var/lib/kubelet/pods/85639f6e-1f74-4502-880c-55da7cf7fa4b/volumes"
Nov 03 21:00:24.142178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:24.142150    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:00:27.380230 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:27.380187    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:00:38.357622 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.357584    4274 generic.go:356] "Generic (PLEG): container finished" podID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerID="b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827" exitCode=0
Nov 03 21:00:38.358079 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.357639    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerDied","Data":"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"}
Nov 03 21:00:38.358160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.358142    4274 scope.go:122] "RemoveContainer" containerID="b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"
Nov 03 21:00:38.359791 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.359754    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:38.359872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.359830    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:38.359872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:38.359861    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:39.389055 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:39.388991    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerStarted","Data":"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"}
Nov 03 21:00:39.673119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:39.669492    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:00:39.677519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:39.677191    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:39.677519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:39.677244    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:40.670440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:40.670399    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:40.670879 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:40.670467    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:41.334694 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:41.334663    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:00:41.670435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:41.670350    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:41.670435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:41.670411    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:42.670507 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:42.670458    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:42.670737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:42.670540    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:43.670389 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:43.670352    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:43.670582 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:43.670415    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:44.670101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:44.670028    4274 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 21:00:44.670527 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:44.670133    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 21:00:45.674634 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:45.674605    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:00:45.675712 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:00:45.675690    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:00:57.366455 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:00:57.366416    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:01:13.072883 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:13.072820    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 21:01:13.074173 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:13.074137    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy" containerID="cri-o://9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4" gracePeriod=30
Nov 03 21:01:16.828076 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:16.828026    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 21:01:16.828572 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:16.828320    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy" containerID="cri-o://a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba" gracePeriod=30
Nov 03 21:01:18.371435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.371404    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 21:01:18.480524 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480480    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.480714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480530    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.480714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480556    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.480714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480613    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.480714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480640    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.480714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480668    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480739    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480772    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480829    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480886    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.480986    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\") pod \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\" (UID: \"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46\") "
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481091    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481208    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481444    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481704    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481727    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.481748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481739    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.482195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481763    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.482195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.481776    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.483051 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.482997    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:18.483132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.483059    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:18.483383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.483355    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.483383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.483377    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz" pod "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" (UID: "d8e49cad-4cd6-4d16-8c2a-8b690c5baa46"). InnerVolumeSpecName "kube-api-access-jmwlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:18.584073 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.584007    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.584073 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.584058    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.584323 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.584076    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwlz\" (UniqueName: \"kubernetes.io/projected/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-kube-api-access-jmwlz\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.584323 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.584122    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.657690 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.657599    4274 generic.go:356] "Generic (PLEG): container finished" podID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerID="9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4" exitCode=0
Nov 03 21:01:18.657690 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.657651    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"
Nov 03 21:01:18.657888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.657693    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" event={"ID":"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46","Type":"ContainerDied","Data":"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"}
Nov 03 21:01:18.657888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.657748    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5" event={"ID":"d8e49cad-4cd6-4d16-8c2a-8b690c5baa46","Type":"ContainerDied","Data":"df7d2ab307fe49158a238a003a1c196ae315653a0413b005e87b336c3a518edb"}
Nov 03 21:01:18.657888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.657771    4274 scope.go:122] "RemoveContainer" containerID="9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"
Nov 03 21:01:18.670428 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.670409    4274 scope.go:122] "RemoveContainer" containerID="9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"
Nov 03 21:01:18.670682 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:01:18.670663    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4\": container with ID starting with 9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4 not found: ID does not exist" containerID="9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"
Nov 03 21:01:18.670741 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.670693    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4"} err="failed to get container status \"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4\": rpc error: code = NotFound desc = could not find container \"9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4\": container with ID starting with 9a86eb23bb7fc370615b13a27cbe1abcbb20a3728bad687227e605d4d02a5bc4 not found: ID does not exist"
Nov 03 21:01:18.813769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.813733    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 21:01:18.824527 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:18.824491    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-qhkx5"]
Nov 03 21:01:19.217068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:19.216463    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" path="/var/lib/kubelet/pods/d8e49cad-4cd6-4d16-8c2a-8b690c5baa46/volumes"
Nov 03 21:01:21.408154 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.408117    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:01:21.409029 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409011    4274 cpu_manager.go:414] "containerMap: removing container" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.409089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409051    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.409089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409073    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409081    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409098    4274 cpu_manager.go:414] "containerMap: removing container" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409106    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409133    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409140    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409150    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy"
Nov 03 21:01:21.409187 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409158    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy"
Nov 03 21:01:21.409361 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409339    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409361 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409351    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e49cad-4cd6-4d16-8c2a-8b690c5baa46" containerName="istio-proxy"
Nov 03 21:01:21.409435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409367    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.409435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409380    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:21.409435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.409393    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:21.414989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.414966    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.466671 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.466641    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:01:21.507977 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.507939    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d78b90\""
Nov 03 21:01:21.508182 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.508056    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d78b90-dockercfg-v8bmk\""
Nov 03 21:01:21.528179 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.528144    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.528683 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.528655    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4kgw\" (UniqueName: \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.642162 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.642111    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kgw\" (UniqueName: \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.642334 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.642261    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.645819 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.645788    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.700098 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.700010    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4kgw\" (UniqueName: \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\") pod \"payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:21.731836 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:21.731802    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:22.121954 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.121922    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:01:22.122857 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:01:22.122829    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d78142_b65b_405f_aaf8_0a5dd2e13114.slice/crio-f219e6f04d52a87d3ba59237a99b94a7b8b2328ee28fa414e2027a4976871da6 WatchSource:0}: Error finding container f219e6f04d52a87d3ba59237a99b94a7b8b2328ee28fa414e2027a4976871da6: Status 404 returned error can't find the container with id f219e6f04d52a87d3ba59237a99b94a7b8b2328ee28fa414e2027a4976871da6
Nov 03 21:01:22.145295 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.145272    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 21:01:22.256767 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256746    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.256887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256799    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.256887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256818    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.256887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256865    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.256887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256888    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.257188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256948    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.257188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.256971    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.257188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257030    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.257188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257110    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.257398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257221    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\") pod \"a1622d3b-4d55-415a-827f-8143ce51b5e5\" (UID: \"a1622d3b-4d55-415a-827f-8143ce51b5e5\") "
Nov 03 21:01:22.257398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257247    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.257502 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257452    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.257725 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257699    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:22.257842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257811    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.257951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257895    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.257951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257917    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1622d3b-4d55-415a-827f-8143ce51b5e5-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.257951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257931    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.257951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.257945    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.259358 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.259333    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.259460 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.259414    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "kube-api-access-h588b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:22.259807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.259779    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:22.259908 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.259849    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token" pod "a1622d3b-4d55-415a-827f-8143ce51b5e5" (UID: "a1622d3b-4d55-415a-827f-8143ce51b5e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:22.359337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.359306    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.359337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.359339    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.359521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.359354    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.359521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.359369    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h588b\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-kube-api-access-h588b\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.359521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.359384    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1622d3b-4d55-415a-827f-8143ce51b5e5-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.687518 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.687426    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" event={"ID":"21d78142-b65b-405f-aaf8-0a5dd2e13114","Type":"ContainerStarted","Data":"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"}
Nov 03 21:01:22.687518 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.687486    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" event={"ID":"21d78142-b65b-405f-aaf8-0a5dd2e13114","Type":"ContainerStarted","Data":"f219e6f04d52a87d3ba59237a99b94a7b8b2328ee28fa414e2027a4976871da6"}
Nov 03 21:01:22.687518 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.687514    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:22.688893 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.688870    4274 generic.go:356] "Generic (PLEG): container finished" podID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerID="a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba" exitCode=0
Nov 03 21:01:22.688998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.688904    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" event={"ID":"a1622d3b-4d55-415a-827f-8143ce51b5e5","Type":"ContainerDied","Data":"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"}
Nov 03 21:01:22.688998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.688912    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"
Nov 03 21:01:22.688998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.688960    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6" event={"ID":"a1622d3b-4d55-415a-827f-8143ce51b5e5","Type":"ContainerDied","Data":"c51bf1d3a93c817a8abdbd226ead9e613b2a35787fa13041a33099447ea248a1"}
Nov 03 21:01:22.688998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.688983    4274 scope.go:122] "RemoveContainer" containerID="a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"
Nov 03 21:01:22.699094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.699072    4274 scope.go:122] "RemoveContainer" containerID="a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"
Nov 03 21:01:22.699343 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:01:22.699328    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba\": container with ID starting with a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba not found: ID does not exist" containerID="a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"
Nov 03 21:01:22.699406 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.699351    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba"} err="failed to get container status \"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba\": rpc error: code = NotFound desc = could not find container \"a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba\": container with ID starting with a7fa4a58530906474eecd0cc9283f139f3bbe5ca1bf4d7c1e152f63daf1d60ba not found: ID does not exist"
Nov 03 21:01:22.798866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.798814    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" podStartSLOduration=1.7988007769999999 podStartE2EDuration="1.798800777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:22.796305644 +0000 UTC m=+1907.722875882" watchObservedRunningTime="2026-11-03 21:01:22.798800777 +0000 UTC m=+1907.725371015"
Nov 03 21:01:22.901748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.901719    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 21:01:22.922082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:22.916179    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-278c6"]
Nov 03 21:01:23.216159 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:23.216127    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" path="/var/lib/kubelet/pods/a1622d3b-4d55-415a-827f-8143ce51b5e5/volumes"
Nov 03 21:01:24.033676 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.033640    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:01:24.034454 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034431    4274 cpu_manager.go:414] "containerMap: removing container" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy"
Nov 03 21:01:24.034583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034455    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy"
Nov 03 21:01:24.034583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034477    4274 cpu_manager.go:414] "containerMap: removing container" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:24.034583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034486    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:24.034583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034512    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:24.034583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034519    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:24.034816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034689    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85639f6e-1f74-4502-880c-55da7cf7fa4b" containerName="cleanup"
Nov 03 21:01:24.034816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034714    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d176d16b-3f6e-4c8d-998a-ad2959b5c099" containerName="cleanup"
Nov 03 21:01:24.034816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.034730    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1622d3b-4d55-415a-827f-8143ce51b5e5" containerName="istio-proxy"
Nov 03 21:01:24.039809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.039785    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.089264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.089237    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8448c9-openshift-default-dockercfg-rszlv\""
Nov 03 21:01:24.141092 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.141061    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:01:24.182829 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.182765    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.182871    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.182928    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183003    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183064    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183105    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183157    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6q6s\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183405 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183290    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.183405 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.183321    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284388    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284426    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284668 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284480    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284668 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284541    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284668 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284590    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284668 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284662    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284876 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284704    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284876 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284739    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.284876 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284787    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6q6s\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.285071 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.284891    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.285211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.285159    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.285298 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.285230    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.285298 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.285256    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.285449 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.285430    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.286785 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.286760    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.287261 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.287231    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.341656 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.341625    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6q6s\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.341656 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.341648    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\") pod \"e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") " pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.352946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.352923    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:24.776991 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:01:24.776956    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629b1544_5969_4684_903c_8923c198ea8f.slice/crio-77b1c711c63873a7f6e435194afbffdac7938d41ebe41cf592ad20b561d2cd9c WatchSource:0}: Error finding container 77b1c711c63873a7f6e435194afbffdac7938d41ebe41cf592ad20b561d2cd9c: Status 404 returned error can't find the container with id 77b1c711c63873a7f6e435194afbffdac7938d41ebe41cf592ad20b561d2cd9c
Nov 03 21:01:24.778544 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.778520    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:01:24.780395 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.780359    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:24.780496 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.780440    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:24.780496 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:24.780485    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:25.710140 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:25.710106    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerStarted","Data":"7e1285f376d1e27d3f36b8fb2d36a801931382d9d988eea3129703bc0855279c"}
Nov 03 21:01:25.710624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:25.710155    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerStarted","Data":"77b1c711c63873a7f6e435194afbffdac7938d41ebe41cf592ad20b561d2cd9c"}
Nov 03 21:01:25.841395 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:25.841346    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podStartSLOduration=2.841333289 podStartE2EDuration="2.841333289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:25.840338745 +0000 UTC m=+1910.766908981" watchObservedRunningTime="2026-11-03 21:01:25.841333289 +0000 UTC m=+1910.767903620"
Nov 03 21:01:26.353586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:26.353553    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:26.359305 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:26.359283    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:26.717769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:26.717688    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:26.718623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:26.718602    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:01:27.366549 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:01:27.366515    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:01:33.699248 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:33.699220    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:01:57.316294 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.316262    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:01:57.320545 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.320521    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.373071 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:01:57.373000    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:01:57.384884 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.384861    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-878695-openshift-default-dockercfg-vzsvg\""
Nov 03 21:01:57.419503 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.419473    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:01:57.481892 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.481842    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.481930    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482002    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482055    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482307 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482169    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482307 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482195    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482423 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482367    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph425\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482421    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.482540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.482525    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587440    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph425\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587513    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587602    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587666    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587763    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587829    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587859    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587912    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.588137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.587936    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.589115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.589055    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.589471 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.589446    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.589702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.589683    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.590375 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.590345    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.592155 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.592129    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.592604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.592582    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.606129 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.596352    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.649982 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.649954    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph425\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.650343 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.650297    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\") pod \"e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") " pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:57.934172 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:57.934096    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:58.408546 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.408517    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:01:58.409167 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:01:58.409124    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cc125f_f1b1_4630_a220_4f5e7631ebac.slice/crio-0c63df998977d82a7be0ff1eff659b2fb6e74270b4487ff75f245ce23d0f738a WatchSource:0}: Error finding container 0c63df998977d82a7be0ff1eff659b2fb6e74270b4487ff75f245ce23d0f738a: Status 404 returned error can't find the container with id 0c63df998977d82a7be0ff1eff659b2fb6e74270b4487ff75f245ce23d0f738a
Nov 03 21:01:58.411858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.411822    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:58.411969 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.411902    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:58.411969 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.411948    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:58.953403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.953362    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerStarted","Data":"ad4aff4849df5aad20fbd9057a734fbe4d81e4a7eb93aaccef4820fde2f575a9"}
Nov 03 21:01:58.953577 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:58.953428    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerStarted","Data":"0c63df998977d82a7be0ff1eff659b2fb6e74270b4487ff75f245ce23d0f738a"}
Nov 03 21:01:59.116330 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:59.116277    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podStartSLOduration=2.116262919 podStartE2EDuration="2.116262919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:59.111903255 +0000 UTC m=+1944.038473493" watchObservedRunningTime="2026-11-03 21:01:59.116262919 +0000 UTC m=+1944.042833155"
Nov 03 21:01:59.934544 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:59.934502    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:59.940156 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:59.940134    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:59.960704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:59.960679    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:01:59.961361 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:01:59.961341    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:02:27.371016 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:02:27.370977    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:02:57.366584 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:02:57.366543    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:03:21.060696 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:21.060610    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 21:03:26.061619 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.061570    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" podUID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" containerName="payload-processing" containerID="cri-o://49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb" gracePeriod=25
Nov 03 21:03:26.334371 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.334346    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 21:03:26.400853 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.400825    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\" (UniqueName: \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\") pod \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") "
Nov 03 21:03:26.401011 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.400925    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\") pod \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\" (UID: \"4bd2eba3-d917-4d84-8d6a-e4687354d8b5\") "
Nov 03 21:03:26.401281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.401257    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume" pod "4bd2eba3-d917-4d84-8d6a-e4687354d8b5" (UID: "4bd2eba3-d917-4d84-8d6a-e4687354d8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:26.401553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.401534    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:26.402955 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.402935    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v" pod "4bd2eba3-d917-4d84-8d6a-e4687354d8b5" (UID: "4bd2eba3-d917-4d84-8d6a-e4687354d8b5"). InnerVolumeSpecName "kube-api-access-dzd8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:26.503333 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.503281    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzd8v\" (UniqueName: \"kubernetes.io/projected/4bd2eba3-d917-4d84-8d6a-e4687354d8b5-kube-api-access-dzd8v\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:26.746024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.745938    4274 generic.go:356] "Generic (PLEG): container finished" podID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" containerID="49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb" exitCode=0
Nov 03 21:03:26.746024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.746005    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"
Nov 03 21:03:26.746024 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.746025    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" event={"ID":"4bd2eba3-d917-4d84-8d6a-e4687354d8b5","Type":"ContainerDied","Data":"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"}
Nov 03 21:03:26.746344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.746107    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l" event={"ID":"4bd2eba3-d917-4d84-8d6a-e4687354d8b5","Type":"ContainerDied","Data":"c2c677f1f786e11d1dee860788d7aabb494068204e12ff1c86f73d60ccc51a03"}
Nov 03 21:03:26.746344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.746134    4274 scope.go:122] "RemoveContainer" containerID="49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"
Nov 03 21:03:26.758978 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.758423    4274 scope.go:122] "RemoveContainer" containerID="49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"
Nov 03 21:03:26.759420 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:03:26.759384    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb\": container with ID starting with 49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb not found: ID does not exist" containerID="49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"
Nov 03 21:03:26.759550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.759428    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb"} err="failed to get container status \"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb\": rpc error: code = NotFound desc = could not find container \"49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb\": container with ID starting with 49d9da0ce3a12be16e87a3da8021976594f8286654b5366d5d8b9941995609fb not found: ID does not exist"
Nov 03 21:03:26.907429 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.904383    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 21:03:26.947064 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:26.947012    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5b12d6-6d6fd8c585-xjl9l"]
Nov 03 21:03:27.215243 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:27.215214    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" path="/var/lib/kubelet/pods/4bd2eba3-d917-4d84-8d6a-e4687354d8b5/volumes"
Nov 03 21:03:27.368063 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:03:27.368011    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:03:38.842357 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.842321    4274 generic.go:356] "Generic (PLEG): container finished" podID="629b1544-5969-4684-903c-8923c198ea8f" containerID="7e1285f376d1e27d3f36b8fb2d36a801931382d9d988eea3129703bc0855279c" exitCode=0
Nov 03 21:03:38.842898 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.842367    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerDied","Data":"7e1285f376d1e27d3f36b8fb2d36a801931382d9d988eea3129703bc0855279c"}
Nov 03 21:03:38.842898 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.842770    4274 scope.go:122] "RemoveContainer" containerID="7e1285f376d1e27d3f36b8fb2d36a801931382d9d988eea3129703bc0855279c"
Nov 03 21:03:38.844513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.844477    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:03:38.844587 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.844550    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:03:38.844632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:38.844584    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:03:39.873278 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:39.873235    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerStarted","Data":"cf5079141c9a8e12102094d13d121d79e5b87d696ae3c32d03f5ecd74d227f5d"}
Nov 03 21:03:40.374063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:40.373438    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:03:40.376058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:40.374346    4274 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 21:03:40.376058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:40.374401    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 21:03:41.370442 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:41.370404    4274 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 21:03:41.371478 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:41.371008    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 21:03:41.718252 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:41.718162    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:03:42.366060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:42.358985    4274 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 21:03:42.366060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:42.359076    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 21:03:43.370166 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:43.370128    4274 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 21:03:43.370786 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:43.370751    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 21:03:44.366445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:44.366406    4274 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 21:03:44.370190 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:44.370138    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 21:03:45.358621 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:45.358590    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:03:45.359583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:45.359560    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:03:47.982293 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:47.982213    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 21:03:47.982678 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:47.982481    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" containerID="cri-o://634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405" gracePeriod=30
Nov 03 21:03:52.018322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:52.018283    4274 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:03:52.018681 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:52.018335    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:03:53.256814 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.256790    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 21:03:53.357871 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.357842    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358077 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.357886    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358077 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.357911    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358077 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358008    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358101    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358163    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358365 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358206    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:53.358365 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358263    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358470 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358423    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:53.358470 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358343    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:53.358557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358472    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:53.358719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358392    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.358923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.358903    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\") pod \"3035b852-0efc-4043-b0f2-ecd926963754\" (UID: \"3035b852-0efc-4043-b0f2-ecd926963754\") "
Nov 03 21:03:53.359478 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.359441    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:53.360113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360091    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.360294 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360280    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.360395 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360381    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.360505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360470    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3035b852-0efc-4043-b0f2-ecd926963754-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.360505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360492    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.360716 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360681    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "kube-api-access-lgqrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:53.360826 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360763    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:53.360826 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360777    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:53.360941 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.360824    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo" pod "3035b852-0efc-4043-b0f2-ecd926963754" (UID: "3035b852-0efc-4043-b0f2-ecd926963754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:53.462344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.462310    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.462344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.462334    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqrn\" (UniqueName: \"kubernetes.io/projected/3035b852-0efc-4043-b0f2-ecd926963754-kube-api-access-lgqrn\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.462344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.462346    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3035b852-0efc-4043-b0f2-ecd926963754-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.462654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.462357    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3035b852-0efc-4043-b0f2-ecd926963754-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:53.965851 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.965818    4274 generic.go:356] "Generic (PLEG): container finished" podID="3035b852-0efc-4043-b0f2-ecd926963754" containerID="634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405" exitCode=0
Nov 03 21:03:53.966074 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.965858    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"
Nov 03 21:03:53.966074 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.965894    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" event={"ID":"3035b852-0efc-4043-b0f2-ecd926963754","Type":"ContainerDied","Data":"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"}
Nov 03 21:03:53.966074 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.965957    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4" event={"ID":"3035b852-0efc-4043-b0f2-ecd926963754","Type":"ContainerDied","Data":"360ffc7e6fbbe0210222276afdcb5005ed94c72c98de8bd8fea33ce9d73aa8f3"}
Nov 03 21:03:53.966074 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.965986    4274 scope.go:122] "RemoveContainer" containerID="634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"
Nov 03 21:03:53.976610 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.976587    4274 scope.go:122] "RemoveContainer" containerID="634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"
Nov 03 21:03:53.976870 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:03:53.976852    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405\": container with ID starting with 634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405 not found: ID does not exist" containerID="634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"
Nov 03 21:03:53.976932 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:53.976876    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405"} err="failed to get container status \"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405\": rpc error: code = NotFound desc = could not find container \"634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405\": container with ID starting with 634e99222c6840e2659a7d54ebdfaf0efdcd3e8e3c8330efaa737818f43f3405 not found: ID does not exist"
Nov 03 21:03:54.120937 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:54.120906    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 21:03:54.166911 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:54.166884    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-f9wg4"]
Nov 03 21:03:55.213809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:55.213774    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3035b852-0efc-4043-b0f2-ecd926963754" path="/var/lib/kubelet/pods/3035b852-0efc-4043-b0f2-ecd926963754/volumes"
Nov 03 21:03:57.367203 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:03:57.367171    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:03:59.217206 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.217172    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 21:03:59.217671 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.217369    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" containerID="cri-o://1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2" gracePeriod=30
Nov 03 21:03:59.262904 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.262872    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 21:03:59.263158 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.263124    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" containerID="cri-o://99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a" gracePeriod=30
Nov 03 21:03:59.266173 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.266148    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 21:03:59.266334 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:03:59.266315    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy" containerID="cri-o://efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e" gracePeriod=30
Nov 03 21:04:02.452572 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:02.452528    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:02.452948 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:02.452590    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:03.792692 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:03.792659    4274 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:03.793106 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:03.792724    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:03.839945 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:03.839902    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 21:04:03.955944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:03.955910    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 21:04:04.456380 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:04.446609    4274 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8969662_12c2_4849_b8a6_4993f5f760ca.slice/crio-conmon-efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8969662_12c2_4849_b8a6_4993f5f760ca.slice/crio-efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:04:04.570439 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.570368    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 21:04:04.623401 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.623372    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 21:04:04.631239 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631211    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631267    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631303    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631325    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631353    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631512    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.631624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631551    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631602    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631779 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631684    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631779 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631715    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\") pod \"f8969662-12c2-4849-b8a6-4993f5f760ca\" (UID: \"f8969662-12c2-4849-b8a6-4993f5f760ca\") "
Nov 03 21:04:04.631925 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.631901    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.632047 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632015    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.632410 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632301    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.632410 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632355    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.632604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632450    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.632604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632470    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.632604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632484    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.632604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632497    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.632604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.632511    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8969662-12c2-4849-b8a6-4993f5f760ca-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.634176 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.634154    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.635114 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.635088    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.635209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.635139    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.635209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.635168    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn" pod "f8969662-12c2-4849-b8a6-4993f5f760ca" (UID: "f8969662-12c2-4849-b8a6-4993f5f760ca"). InnerVolumeSpecName "kube-api-access-c2gmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.663945 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.663920    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 21:04:04.733468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.733436    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.733719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.733539    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.733801 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.733578    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.733910 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.733888    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734282 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734260    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734640 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734618    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734762 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734673    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734833 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734815    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734860    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.734936 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734886    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.734986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734940    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.734986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.734976    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\") pod \"802b6f5e-10d7-46ce-8c44-3eb446983e13\" (UID: \"802b6f5e-10d7-46ce-8c44-3eb446983e13\") "
Nov 03 21:04:04.735132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735023    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.735132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735077    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.735132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735113    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.735286 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735251    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.735337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735294    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.735337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.735329    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\") pod \"1606d4c8-0342-47b2-8258-d5f019552a36\" (UID: \"1606d4c8-0342-47b2-8258-d5f019552a36\") "
Nov 03 21:04:04.736166 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736138    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.736259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736172    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gmn\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-kube-api-access-c2gmn\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.736259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736187    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.736259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736200    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8969662-12c2-4849-b8a6-4993f5f760ca-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.736462 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736438    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736666    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.736978    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737247    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737349    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "kube-api-access-x4xnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737549    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737555    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737607    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737663    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.737850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737685    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.738634 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.737912    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.738634 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.738400    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.738955 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.738894    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.739909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.739875    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.740018 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.739997    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.740154 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.739999    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.741164 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.741132    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo" pod "1606d4c8-0342-47b2-8258-d5f019552a36" (UID: "1606d4c8-0342-47b2-8258-d5f019552a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.741405 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.741368    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc" pod "802b6f5e-10d7-46ce-8c44-3eb446983e13" (UID: "802b6f5e-10d7-46ce-8c44-3eb446983e13"). InnerVolumeSpecName "kube-api-access-qffpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837605    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1606d4c8-0342-47b2-8258-d5f019552a36-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837647    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837662    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837678    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xnf\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-kube-api-access-x4xnf\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837699    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837713    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837726    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffpc\" (UniqueName: \"kubernetes.io/projected/802b6f5e-10d7-46ce-8c44-3eb446983e13-kube-api-access-qffpc\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837739    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837754    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802b6f5e-10d7-46ce-8c44-3eb446983e13-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837766    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837778    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837795    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837807    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1606d4c8-0342-47b2-8258-d5f019552a36-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837819    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837831    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802b6f5e-10d7-46ce-8c44-3eb446983e13-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837846    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837858    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1606d4c8-0342-47b2-8258-d5f019552a36-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.837989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:04.837870    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1606d4c8-0342-47b2-8258-d5f019552a36-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:05.039602 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.039570    4274 generic.go:356] "Generic (PLEG): container finished" podID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerID="1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2" exitCode=0
Nov 03 21:04:05.039789 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.039612    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerDied","Data":"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"}
Nov 03 21:04:05.039789 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.039636    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"
Nov 03 21:04:05.039789 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.039676    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8" event={"ID":"802b6f5e-10d7-46ce-8c44-3eb446983e13","Type":"ContainerDied","Data":"7a5c3ea4a2bd8efde7f43f7b526b9183f7504b2dcf148db974ce626bca8258f1"}
Nov 03 21:04:05.039789 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.039703    4274 scope.go:122] "RemoveContainer" containerID="1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"
Nov 03 21:04:05.041279 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.041254    4274 generic.go:356] "Generic (PLEG): container finished" podID="1606d4c8-0342-47b2-8258-d5f019552a36" containerID="99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a" exitCode=0
Nov 03 21:04:05.041403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.041297    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"
Nov 03 21:04:05.041403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.041296    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" event={"ID":"1606d4c8-0342-47b2-8258-d5f019552a36","Type":"ContainerDied","Data":"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"}
Nov 03 21:04:05.041403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.041354    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf" event={"ID":"1606d4c8-0342-47b2-8258-d5f019552a36","Type":"ContainerDied","Data":"af2b873b1a03a4681e204ecc839a0407aca0d402d59d5aba96de368e38248450"}
Nov 03 21:04:05.043029 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.042997    4274 generic.go:356] "Generic (PLEG): container finished" podID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerID="efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e" exitCode=0
Nov 03 21:04:05.043029 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.043024    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"
Nov 03 21:04:05.043241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.043082    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" event={"ID":"f8969662-12c2-4849-b8a6-4993f5f760ca","Type":"ContainerDied","Data":"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"}
Nov 03 21:04:05.043241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.043129    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm" event={"ID":"f8969662-12c2-4849-b8a6-4993f5f760ca","Type":"ContainerDied","Data":"c63b1fc3b2cf46e3fe8708d8c3e7e8097ac52059c76000234f8a307232ec6005"}
Nov 03 21:04:05.052120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.052107    4274 scope.go:122] "RemoveContainer" containerID="dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"
Nov 03 21:04:05.062839 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.062777    4274 scope.go:122] "RemoveContainer" containerID="1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"
Nov 03 21:04:05.063151 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:05.063115    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2\": container with ID starting with 1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2 not found: ID does not exist" containerID="1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"
Nov 03 21:04:05.063270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.063155    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2"} err="failed to get container status \"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2\": rpc error: code = NotFound desc = could not find container \"1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2\": container with ID starting with 1ba739b2741578d3b02e4fbe02ba013d195f568e59fdab779c89d03a82d8c3f2 not found: ID does not exist"
Nov 03 21:04:05.063270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.063181    4274 scope.go:122] "RemoveContainer" containerID="dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"
Nov 03 21:04:05.063900 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:05.063590    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe\": container with ID starting with dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe not found: ID does not exist" containerID="dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"
Nov 03 21:04:05.063900 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.063662    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe"} err="failed to get container status \"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe\": rpc error: code = NotFound desc = could not find container \"dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe\": container with ID starting with dc27da6e0ed5cb1d15261540bbfa9625eaa222479dcee8bd90c3bb5c9c5151fe not found: ID does not exist"
Nov 03 21:04:05.063900 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.063742    4274 scope.go:122] "RemoveContainer" containerID="99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"
Nov 03 21:04:05.098435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.097626    4274 scope.go:122] "RemoveContainer" containerID="99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"
Nov 03 21:04:05.098435 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:05.098258    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a\": container with ID starting with 99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a not found: ID does not exist" containerID="99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"
Nov 03 21:04:05.098435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.098318    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a"} err="failed to get container status \"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a\": rpc error: code = NotFound desc = could not find container \"99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a\": container with ID starting with 99c5bfaa7d5f0de0878aefe7442fddaca39b9f5c539fbd3279a5b6b8b3d8c79a not found: ID does not exist"
Nov 03 21:04:05.098435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.098340    4274 scope.go:122] "RemoveContainer" containerID="efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"
Nov 03 21:04:05.108874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.108854    4274 scope.go:122] "RemoveContainer" containerID="efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"
Nov 03 21:04:05.109284 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:05.109256    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e\": container with ID starting with efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e not found: ID does not exist" containerID="efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"
Nov 03 21:04:05.109394 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.109291    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e"} err="failed to get container status \"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e\": rpc error: code = NotFound desc = could not find container \"efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e\": container with ID starting with efbb24ccb53666f5831ef8cb170ae63bedff915e431c6281900d92ef25380c9e not found: ID does not exist"
Nov 03 21:04:05.204423 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.204387    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 21:04:05.213951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.213926    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rc7hf"]
Nov 03 21:04:05.314134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.314103    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 21:04:05.317383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.317359    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-rkzdm"]
Nov 03 21:04:05.437113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.423832    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 21:04:05.437113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:05.428924    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-q4cq8"]
Nov 03 21:04:07.213832 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:07.213798    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" path="/var/lib/kubelet/pods/1606d4c8-0342-47b2-8258-d5f019552a36/volumes"
Nov 03 21:04:07.214302 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:07.214287    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" path="/var/lib/kubelet/pods/802b6f5e-10d7-46ce-8c44-3eb446983e13/volumes"
Nov 03 21:04:07.214740 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:07.214727    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" path="/var/lib/kubelet/pods/f8969662-12c2-4849-b8a6-4993f5f760ca/volumes"
Nov 03 21:04:08.841113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:08.841064    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" podUID="bae534ad-9d21-4d8a-88be-b66d56b08eef" containerName="payload-processing" containerID="cri-o://a9fbaf8f60f7b30b7a5bf68a1082e0c26eb4dd98c1571b94184a21265901fb05" gracePeriod=25
Nov 03 21:04:08.957163 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:08.957130    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" podUID="4f00c265-f625-4797-834f-cacad9b68ab0" containerName="payload-pre-processing" containerID="cri-o://9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103" gracePeriod=25
Nov 03 21:04:09.073327 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.073140    4274 generic.go:356] "Generic (PLEG): container finished" podID="bae534ad-9d21-4d8a-88be-b66d56b08eef" containerID="a9fbaf8f60f7b30b7a5bf68a1082e0c26eb4dd98c1571b94184a21265901fb05" exitCode=0
Nov 03 21:04:09.073327 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.073184    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" event={"ID":"bae534ad-9d21-4d8a-88be-b66d56b08eef","Type":"ContainerDied","Data":"a9fbaf8f60f7b30b7a5bf68a1082e0c26eb4dd98c1571b94184a21265901fb05"}
Nov 03 21:04:09.114132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.111784    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 21:04:09.198435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.198405    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\" (UniqueName: \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\") pod \"bae534ad-9d21-4d8a-88be-b66d56b08eef\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") "
Nov 03 21:04:09.198615 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.198507    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\") pod \"bae534ad-9d21-4d8a-88be-b66d56b08eef\" (UID: \"bae534ad-9d21-4d8a-88be-b66d56b08eef\") "
Nov 03 21:04:09.198957 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.198931    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume" pod "bae534ad-9d21-4d8a-88be-b66d56b08eef" (UID: "bae534ad-9d21-4d8a-88be-b66d56b08eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:09.199170 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.199152    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae534ad-9d21-4d8a-88be-b66d56b08eef-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:09.200623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.200594    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt" pod "bae534ad-9d21-4d8a-88be-b66d56b08eef" (UID: "bae534ad-9d21-4d8a-88be-b66d56b08eef"). InnerVolumeSpecName "kube-api-access-7scxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:09.225884 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.225432    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 21:04:09.300223 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.300194    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\") pod \"4f00c265-f625-4797-834f-cacad9b68ab0\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") "
Nov 03 21:04:09.300424 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.300410    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\" (UniqueName: \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\") pod \"4f00c265-f625-4797-834f-cacad9b68ab0\" (UID: \"4f00c265-f625-4797-834f-cacad9b68ab0\") "
Nov 03 21:04:09.300576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.300550    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume" pod "4f00c265-f625-4797-834f-cacad9b68ab0" (UID: "4f00c265-f625-4797-834f-cacad9b68ab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:09.300811 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.300796    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f00c265-f625-4797-834f-cacad9b68ab0-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:09.300869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.300812    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scxt\" (UniqueName: \"kubernetes.io/projected/bae534ad-9d21-4d8a-88be-b66d56b08eef-kube-api-access-7scxt\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:09.302552 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.302527    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm" pod "4f00c265-f625-4797-834f-cacad9b68ab0" (UID: "4f00c265-f625-4797-834f-cacad9b68ab0"). InnerVolumeSpecName "kube-api-access-v7lxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:09.402116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:09.402009    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lxm\" (UniqueName: \"kubernetes.io/projected/4f00c265-f625-4797-834f-cacad9b68ab0-kube-api-access-v7lxm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:10.081798 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.081767    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"
Nov 03 21:04:10.082331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.081776    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z" event={"ID":"bae534ad-9d21-4d8a-88be-b66d56b08eef","Type":"ContainerDied","Data":"ee6522b46abf6a5e2e54871ebe190ffc75d8ab46b2450a4653829d1dd9d78c45"}
Nov 03 21:04:10.082331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.081910    4274 scope.go:122] "RemoveContainer" containerID="a9fbaf8f60f7b30b7a5bf68a1082e0c26eb4dd98c1571b94184a21265901fb05"
Nov 03 21:04:10.083376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.083168    4274 generic.go:356] "Generic (PLEG): container finished" podID="4f00c265-f625-4797-834f-cacad9b68ab0" containerID="9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103" exitCode=0
Nov 03 21:04:10.083376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.083246    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"
Nov 03 21:04:10.083376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.083250    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" event={"ID":"4f00c265-f625-4797-834f-cacad9b68ab0","Type":"ContainerDied","Data":"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"}
Nov 03 21:04:10.083549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.083373    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw" event={"ID":"4f00c265-f625-4797-834f-cacad9b68ab0","Type":"ContainerDied","Data":"bcb438363c07f4e33ebb20c91aa08e58a85ac1cd8c519b2fb1513319990d3339"}
Nov 03 21:04:10.092928 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.092913    4274 scope.go:122] "RemoveContainer" containerID="9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"
Nov 03 21:04:10.101434 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.101418    4274 scope.go:122] "RemoveContainer" containerID="9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"
Nov 03 21:04:10.101705 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:10.101687    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103\": container with ID starting with 9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103 not found: ID does not exist" containerID="9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"
Nov 03 21:04:10.101749 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.101713    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103"} err="failed to get container status \"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103\": rpc error: code = NotFound desc = could not find container \"9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103\": container with ID starting with 9aeab51e1dc2bf524c10d5f7912eabcbda0683d80c495dc0ac37869ade3a9103 not found: ID does not exist"
Nov 03 21:04:10.239139 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.239062    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 21:04:10.285297 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.285263    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ba7e83-6dc564944c-rqm9z"]
Nov 03 21:04:10.350507 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.350472    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 21:04:10.356514 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:10.356483    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ba7e83-5fc994cd99-wwtjw"]
Nov 03 21:04:11.218090 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:11.218052    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f00c265-f625-4797-834f-cacad9b68ab0" path="/var/lib/kubelet/pods/4f00c265-f625-4797-834f-cacad9b68ab0/volumes"
Nov 03 21:04:11.218570 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:11.218546    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae534ad-9d21-4d8a-88be-b66d56b08eef" path="/var/lib/kubelet/pods/bae534ad-9d21-4d8a-88be-b66d56b08eef/volumes"
Nov 03 21:04:14.323933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:14.323899    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 21:04:14.324703 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:14.324106    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" containerID="cri-o://b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59" gracePeriod=30
Nov 03 21:04:18.224217 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:18.224180    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:18.224608 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:18.224235    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:18.703694 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:18.703660    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 21:04:18.703885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:18.703864    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" podUID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" containerName="maas-api" containerID="cri-o://dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a" gracePeriod=30
Nov 03 21:04:19.110689 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.110662    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 21:04:19.139728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.139704    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") pod \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") "
Nov 03 21:04:19.139876 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.139744    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\" (UniqueName: \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\") pod \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") "
Nov 03 21:04:19.140313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.140265    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\") pod \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") "
Nov 03 21:04:19.140631 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.140590    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\") pod \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\" (UID: \"4fa3748f-ee00-48c0-bfc2-29fec5e25727\") "
Nov 03 21:04:19.140955 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.140928    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca" pod "4fa3748f-ee00-48c0-bfc2-29fec5e25727" (UID: "4fa3748f-ee00-48c0-bfc2-29fec5e25727"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:19.141731 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.141709    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fa3748f-ee00-48c0-bfc2-29fec5e25727-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.142494 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.142473    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs" pod "4fa3748f-ee00-48c0-bfc2-29fec5e25727" (UID: "4fa3748f-ee00-48c0-bfc2-29fec5e25727"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:19.142611 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.142499    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv" pod "4fa3748f-ee00-48c0-bfc2-29fec5e25727" (UID: "4fa3748f-ee00-48c0-bfc2-29fec5e25727"). InnerVolumeSpecName "kube-api-access-rj2cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:19.142611 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.142524    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls" pod "4fa3748f-ee00-48c0-bfc2-29fec5e25727" (UID: "4fa3748f-ee00-48c0-bfc2-29fec5e25727"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:19.212058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.211617    4274 generic.go:356] "Generic (PLEG): container finished" podID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" containerID="dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a" exitCode=0
Nov 03 21:04:19.212058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.211882    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.241533    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" event={"ID":"4fa3748f-ee00-48c0-bfc2-29fec5e25727","Type":"ContainerDied","Data":"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"}
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.241588    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74" event={"ID":"4fa3748f-ee00-48c0-bfc2-29fec5e25727","Type":"ContainerDied","Data":"4c4ef2193fe1bd3a37a1da51b02f62f57662732891a73341464f58fe7ae6753c"}
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.241623    4274 scope.go:122] "RemoveContainer" containerID="dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.244558    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.244586    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2cv\" (UniqueName: \"kubernetes.io/projected/4fa3748f-ee00-48c0-bfc2-29fec5e25727-kube-api-access-rj2cv\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.245157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.244603    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fa3748f-ee00-48c0-bfc2-29fec5e25727-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.301773 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.301704    4274 scope.go:122] "RemoveContainer" containerID="dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"
Nov 03 21:04:19.302099 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:19.302026    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a\": container with ID starting with dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a not found: ID does not exist" containerID="dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"
Nov 03 21:04:19.302099 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.302069    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a"} err="failed to get container status \"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a\": rpc error: code = NotFound desc = could not find container \"dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a\": container with ID starting with dd5b9e0da8790bc6a5642ba4625e0661d409517d94f94d16a36b379a1705dc3a not found: ID does not exist"
Nov 03 21:04:19.412108 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.412080    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 21:04:19.415076 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.415046    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5b12d6-56c7b8cd9b-ftn74"]
Nov 03 21:04:19.600936 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.600906    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 21:04:19.680877 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.680845    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.680877 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.680882    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.680923    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.680969    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681087    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681128    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.681299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681163    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681193    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681274    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.681516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681335    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.681516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681367    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681402    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\") pod \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\" (UID: \"8ed4bbe3-f822-4ee1-8adc-24cce400261e\") "
Nov 03 21:04:19.681516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681416    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:19.681722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.681703    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.682022 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.682001    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.682022 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.682024    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.682215 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.682052    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.682215 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.682065    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.682215 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.682078    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.683482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.683458    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.683566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.683495    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "kube-api-access-kmbl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:19.683777 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.683753    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:19.683777 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.683758    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token" pod "8ed4bbe3-f822-4ee1-8adc-24cce400261e" (UID: "8ed4bbe3-f822-4ee1-8adc-24cce400261e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:19.822878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.785169    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.822878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.785212    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbl5\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-kube-api-access-kmbl5\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.822878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.785231    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.822878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:19.785245    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed4bbe3-f822-4ee1-8adc-24cce400261e-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:20.221269 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.221188    4274 generic.go:356] "Generic (PLEG): container finished" podID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerID="b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59" exitCode=0
Nov 03 21:04:20.221269 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.221229    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"
Nov 03 21:04:20.221452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.221278    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" event={"ID":"8ed4bbe3-f822-4ee1-8adc-24cce400261e","Type":"ContainerDied","Data":"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"}
Nov 03 21:04:20.221452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.221329    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2" event={"ID":"8ed4bbe3-f822-4ee1-8adc-24cce400261e","Type":"ContainerDied","Data":"0d164a7c39a89960e9a75b50def09ca453835fe8d903e7624749a1bf15eee346"}
Nov 03 21:04:20.221452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.221354    4274 scope.go:122] "RemoveContainer" containerID="b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"
Nov 03 21:04:20.231539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.231525    4274 scope.go:122] "RemoveContainer" containerID="b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"
Nov 03 21:04:20.231795 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:20.231775    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59\": container with ID starting with b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59 not found: ID does not exist" containerID="b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"
Nov 03 21:04:20.231832 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.231803    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59"} err="failed to get container status \"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59\": rpc error: code = NotFound desc = could not find container \"b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59\": container with ID starting with b91d9dff891995e7abbbfd58cc934b94c59ef3a2b86fefdf18ac6bce4bf4db59 not found: ID does not exist"
Nov 03 21:04:20.376688 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.376621    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 21:04:20.422402 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:20.422369    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-f6xd2"]
Nov 03 21:04:21.214732 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:21.214698    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" path="/var/lib/kubelet/pods/4fa3748f-ee00-48c0-bfc2-29fec5e25727/volumes"
Nov 03 21:04:21.215196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:21.215181    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" path="/var/lib/kubelet/pods/8ed4bbe3-f822-4ee1-8adc-24cce400261e/volumes"
Nov 03 21:04:27.265357 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.265325    4274 generic.go:356] "Generic (PLEG): container finished" podID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerID="ad4aff4849df5aad20fbd9057a734fbe4d81e4a7eb93aaccef4820fde2f575a9" exitCode=0
Nov 03 21:04:27.265785 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.265404    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerDied","Data":"ad4aff4849df5aad20fbd9057a734fbe4d81e4a7eb93aaccef4820fde2f575a9"}
Nov 03 21:04:27.265849 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.265826    4274 scope.go:122] "RemoveContainer" containerID="ad4aff4849df5aad20fbd9057a734fbe4d81e4a7eb93aaccef4820fde2f575a9"
Nov 03 21:04:27.267441 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.267415    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:27.267520 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.267480    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:27.267566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:27.267535    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:27.365794 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:27.365764    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:04:28.289059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:28.287317    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerStarted","Data":"b3a4dce4d203dd1617a37e25bc29fdf0a02141946ef7e514e771e3205fc7bc17"}
Nov 03 21:04:28.934535 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:28.934498    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:04:28.935556 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:28.935532    4274 patch_prober.go:29] interesting pod/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:04:28.935629 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:28.935586    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 21:04:29.472380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.472343    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 21:04:29.472772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.472529    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" containerID="cri-o://83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce" gracePeriod=30
Nov 03 21:04:29.586806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.586772    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"]
Nov 03 21:04:29.587383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587362    4274 cpu_manager.go:414] "containerMap: removing container" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy"
Nov 03 21:04:29.587383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587381    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587392    4274 cpu_manager.go:414] "containerMap: removing container" podUID="bae534ad-9d21-4d8a-88be-b66d56b08eef" containerName="payload-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587400    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae534ad-9d21-4d8a-88be-b66d56b08eef" containerName="payload-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587416    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" containerName="payload-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587422    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" containerName="payload-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587430    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4f00c265-f625-4797-834f-cacad9b68ab0" containerName="payload-pre-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587435    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f00c265-f625-4797-834f-cacad9b68ab0" containerName="payload-pre-processing"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587458    4274 cpu_manager.go:414] "containerMap: removing container" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587465    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587482    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" containerName="maas-api"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587490    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" containerName="maas-api"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587507    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587514    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587533    4274 cpu_manager.go:414] "containerMap: removing container" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587540    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587556    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587564    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587577    4274 cpu_manager.go:414] "containerMap: removing container" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587583    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587710    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8969662-12c2-4849-b8a6-4993f5f760ca" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587723    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587729    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3035b852-0efc-4043-b0f2-ecd926963754" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587736    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed4bbe3-f822-4ee1-8adc-24cce400261e" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587743    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1606d4c8-0342-47b2-8258-d5f019552a36" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587749    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae534ad-9d21-4d8a-88be-b66d56b08eef" containerName="payload-processing"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587756    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd2eba3-d917-4d84-8d6a-e4687354d8b5" containerName="payload-processing"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587763    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f00c265-f625-4797-834f-cacad9b68ab0" containerName="payload-pre-processing"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587773    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="802b6f5e-10d7-46ce-8c44-3eb446983e13" containerName="istio-proxy"
Nov 03 21:04:29.587987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.587779    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa3748f-ee00-48c0-bfc2-29fec5e25727" containerName="maas-api"
Nov 03 21:04:29.591211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.591193    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.677917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.677335    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:29.677917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.677486    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:04:29.677917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.677672    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"]
Nov 03 21:04:29.770987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.770937    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.771240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.771013    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.771309 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.771236    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvwf\" (UniqueName: \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.872604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.872570    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.872785 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.872680    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvwf\" (UniqueName: \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.872863 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.872846    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.873452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.873421    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.875338 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.875312    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.930026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.929998    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvwf\" (UniqueName: \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:29.935006 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.934986    4274 patch_prober.go:29] interesting pod/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:04:29.935107 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.935065    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 21:04:29.965058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:29.963146    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:04:30.205248 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:30.205172    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:30.600284 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:30.600255    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"]
Nov 03 21:04:30.935561 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:30.934753    4274 patch_prober.go:29] interesting pod/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:04:30.935561 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:30.934821    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 21:04:31.310018 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:31.309983    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588" event={"ID":"4fdabddd-5498-4943-b50a-def73d96dfb3","Type":"ContainerStarted","Data":"0fabd614d71c91f427910eba94d739829bca1e1f81e50aa9cabc6b97c4b19e84"}
Nov 03 21:04:31.310230 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:31.310058    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588" event={"ID":"4fdabddd-5498-4943-b50a-def73d96dfb3","Type":"ContainerStarted","Data":"37a6e6cf2fe9eb30b295bc1c9aeaf5a9d40c7f72e32462a2fe5aa59d213baeb2"}
Nov 03 21:04:31.431325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:31.431272    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588" podStartSLOduration=2.43125796 podStartE2EDuration="2.43125796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:31.428715952 +0000 UTC m=+2096.355286189" watchObservedRunningTime="2026-11-03 21:04:31.43125796 +0000 UTC m=+2096.357828196"
Nov 03 21:04:31.934686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:31.934650    4274 patch_prober.go:29] interesting pod/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:04:31.935186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:31.934705    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 21:04:32.320477 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:32.320445    4274 generic.go:356] "Generic (PLEG): container finished" podID="4fdabddd-5498-4943-b50a-def73d96dfb3" containerID="0fabd614d71c91f427910eba94d739829bca1e1f81e50aa9cabc6b97c4b19e84" exitCode=0
Nov 03 21:04:32.320677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:32.320529    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588" event={"ID":"4fdabddd-5498-4943-b50a-def73d96dfb3","Type":"ContainerDied","Data":"0fabd614d71c91f427910eba94d739829bca1e1f81e50aa9cabc6b97c4b19e84"}
Nov 03 21:04:32.936245 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:32.936140    4274 patch_prober.go:29] interesting pod/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 21:04:32.936245 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:32.936200    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 21:04:33.036608 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.036576    4274 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:33.036769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.036636    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:33.566879 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.566844    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:33.622454 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.622365    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\" (UniqueName: \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\") pod \"4fdabddd-5498-4943-b50a-def73d96dfb3\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") "
Nov 03 21:04:33.622629 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.622484    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\") pod \"4fdabddd-5498-4943-b50a-def73d96dfb3\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") "
Nov 03 21:04:33.622629 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.622519    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\") pod \"4fdabddd-5498-4943-b50a-def73d96dfb3\" (UID: \"4fdabddd-5498-4943-b50a-def73d96dfb3\") "
Nov 03 21:04:33.622985 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.622959    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca" pod "4fdabddd-5498-4943-b50a-def73d96dfb3" (UID: "4fdabddd-5498-4943-b50a-def73d96dfb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:33.623110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.623095    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-openshift-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:33.624511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.624482    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf" pod "4fdabddd-5498-4943-b50a-def73d96dfb3" (UID: "4fdabddd-5498-4943-b50a-def73d96dfb3"). InnerVolumeSpecName "kube-api-access-dnvwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:33.625093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.625065    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca" pod "4fdabddd-5498-4943-b50a-def73d96dfb3" (UID: "4fdabddd-5498-4943-b50a-def73d96dfb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:33.724212 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.724176    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fdabddd-5498-4943-b50a-def73d96dfb3-maas-api-xks-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:33.724212 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.724204    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvwf\" (UniqueName: \"kubernetes.io/projected/4fdabddd-5498-4943-b50a-def73d96dfb3-kube-api-access-dnvwf\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:33.939865 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.939787    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:04:33.940936 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:33.940914    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:04:34.341199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.341171    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"
Nov 03 21:04:34.341475 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.341160    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588" event={"ID":"4fdabddd-5498-4943-b50a-def73d96dfb3","Type":"ContainerDied","Data":"37a6e6cf2fe9eb30b295bc1c9aeaf5a9d40c7f72e32462a2fe5aa59d213baeb2"}
Nov 03 21:04:34.341703 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.341500    4274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a6e6cf2fe9eb30b295bc1c9aeaf5a9d40c7f72e32462a2fe5aa59d213baeb2"
Nov 03 21:04:34.811979 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.811955    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 21:04:34.838211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838183    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838263    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838293    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838356    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838384    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838556    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.838718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838615    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838648    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838651    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.838718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838674    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838939 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838781    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\") pod \"07127ca8-63d9-4854-807b-3c19ad32238b\" (UID: \"07127ca8-63d9-4854-807b-3c19ad32238b\") "
Nov 03 21:04:34.838939 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838864    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.838939 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.838880    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.839093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839053    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:34.839347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839327    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.839347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839347    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.839505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839358    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.839505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839366    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.839505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.839378    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07127ca8-63d9-4854-807b-3c19ad32238b-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.840419 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.840401    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:34.840808 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.840786    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:34.840893 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.840812    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "kube-api-access-tl88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:34.840933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.840908    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy" pod "07127ca8-63d9-4854-807b-3c19ad32238b" (UID: "07127ca8-63d9-4854-807b-3c19ad32238b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.941093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.941051    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07127ca8-63d9-4854-807b-3c19ad32238b-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.941093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.941091    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.941627 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.941107    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07127ca8-63d9-4854-807b-3c19ad32238b-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.941627 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:34.941122    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl88s\" (UniqueName: \"kubernetes.io/projected/07127ca8-63d9-4854-807b-3c19ad32238b-kube-api-access-tl88s\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.347932 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.347901    4274 generic.go:356] "Generic (PLEG): container finished" podID="07127ca8-63d9-4854-807b-3c19ad32238b" containerID="83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce" exitCode=0
Nov 03 21:04:35.348092 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.347961    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"
Nov 03 21:04:35.348092 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.347963    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" event={"ID":"07127ca8-63d9-4854-807b-3c19ad32238b","Type":"ContainerDied","Data":"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"}
Nov 03 21:04:35.348195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.348094    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b" event={"ID":"07127ca8-63d9-4854-807b-3c19ad32238b","Type":"ContainerDied","Data":"2bd3c3c0f46a708fc848598e2f1668602b00a599659b2a2b60c2b0622a26e5dd"}
Nov 03 21:04:35.348195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.348116    4274 scope.go:122] "RemoveContainer" containerID="83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"
Nov 03 21:04:35.359829 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.359810    4274 scope.go:122] "RemoveContainer" containerID="83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"
Nov 03 21:04:35.360128 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:35.360104    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce\": container with ID starting with 83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce not found: ID does not exist" containerID="83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"
Nov 03 21:04:35.360205 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.360135    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce"} err="failed to get container status \"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce\": rpc error: code = NotFound desc = could not find container \"83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce\": container with ID starting with 83031dba6038257399d9bec7b9e41ec69eb1dc531d4ee309ada7ddb3b30d28ce not found: ID does not exist"
Nov 03 21:04:35.502092 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.501979    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 21:04:35.509099 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:35.509071    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-n5w6b"]
Nov 03 21:04:37.215982 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:04:37.215950    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" path="/var/lib/kubelet/pods/07127ca8-63d9-4854-807b-3c19ad32238b/volumes"
Nov 03 21:04:57.366187 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:04:57.366152    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:05:07.118933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.118901    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:05:07.119517 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119488    4274 cpu_manager.go:414] "containerMap: removing container" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy"
Nov 03 21:05:07.119517 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119501    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy"
Nov 03 21:05:07.119517 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119514    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4fdabddd-5498-4943-b50a-def73d96dfb3" containerName="revoke-keys"
Nov 03 21:05:07.119517 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119519    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fdabddd-5498-4943-b50a-def73d96dfb3" containerName="revoke-keys"
Nov 03 21:05:07.119755 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119656    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="07127ca8-63d9-4854-807b-3c19ad32238b" containerName="istio-proxy"
Nov 03 21:05:07.119755 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.119672    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fdabddd-5498-4943-b50a-def73d96dfb3" containerName="revoke-keys"
Nov 03 21:05:07.123270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.123252    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.152721 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.152695    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.152871 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.152758    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.152939 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.152921    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153129    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153181    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153288    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153366 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153330    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q79mz\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153424 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153408    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.153462 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.153448    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.172009 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.171985    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-df71d7-openshift-default-dockercfg-mm266\""
Nov 03 21:05:07.215096 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.215064    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:05:07.266345 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.266271    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279050    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279128    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279157    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q79mz\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279208    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279249    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279338    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279406    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.279485    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.276374    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.281077    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.281337    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.295056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.281562    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.305119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.303501    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.333375 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.333337    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.341778 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.341747    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.345189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.344580    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.347273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.346847    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q79mz\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\") pod \"e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") " pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.437085 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.436997    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:07.942942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.942902    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:05:07.961443 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:05:07.961418    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d0a315_aaf8_4a5e_abf5_345fd81b5bea.slice/crio-44c0afc18d6238072d031eb14e25545f0839b31e6fd2b1d785af555bd177b123 WatchSource:0}: Error finding container 44c0afc18d6238072d031eb14e25545f0839b31e6fd2b1d785af555bd177b123: Status 404 returned error can't find the container with id 44c0afc18d6238072d031eb14e25545f0839b31e6fd2b1d785af555bd177b123
Nov 03 21:05:07.963979 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.963944    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:07.964123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.964023    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:07.964123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:07.964110    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:08.576476 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:08.576441    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerStarted","Data":"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"}
Nov 03 21:05:08.576878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:08.576498    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerStarted","Data":"44c0afc18d6238072d031eb14e25545f0839b31e6fd2b1d785af555bd177b123"}
Nov 03 21:05:08.741717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:08.741648    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podStartSLOduration=1.74162961 podStartE2EDuration="1.74162961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:08.735895243 +0000 UTC m=+2133.662465479" watchObservedRunningTime="2026-11-03 21:05:08.74162961 +0000 UTC m=+2133.668199848"
Nov 03 21:05:09.437502 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:09.437461    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:09.442914 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:09.442887    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:09.583095 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:09.583065    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:09.584003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:09.583984    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:05:15.507581 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.505308    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 21:05:15.507581 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.505874    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" podUID="14d269e6-cdb0-4609-9c74-48c461f40bcc" containerName="maas-api" containerID="cri-o://bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8" gracePeriod=30
Nov 03 21:05:15.848113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.848083    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 21:05:15.907664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.907615    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\" (UniqueName: \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\") pod \"14d269e6-cdb0-4609-9c74-48c461f40bcc\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") "
Nov 03 21:05:15.907883 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.907812    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") pod \"14d269e6-cdb0-4609-9c74-48c461f40bcc\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") "
Nov 03 21:05:15.908109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.908059    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\") pod \"14d269e6-cdb0-4609-9c74-48c461f40bcc\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") "
Nov 03 21:05:15.908256 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.908117    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\") pod \"14d269e6-cdb0-4609-9c74-48c461f40bcc\" (UID: \"14d269e6-cdb0-4609-9c74-48c461f40bcc\") "
Nov 03 21:05:15.909314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.909290    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca" pod "14d269e6-cdb0-4609-9c74-48c461f40bcc" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:15.910583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.910551    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls" pod "14d269e6-cdb0-4609-9c74-48c461f40bcc" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:15.910730 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.910699    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs" pod "14d269e6-cdb0-4609-9c74-48c461f40bcc" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:15.910730 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:15.910723    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw" pod "14d269e6-cdb0-4609-9c74-48c461f40bcc" (UID: "14d269e6-cdb0-4609-9c74-48c461f40bcc"). InnerVolumeSpecName "kube-api-access-glgbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:16.010347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.010315    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d269e6-cdb0-4609-9c74-48c461f40bcc-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:16.010347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.010342    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:16.010347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.010353    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glgbw\" (UniqueName: \"kubernetes.io/projected/14d269e6-cdb0-4609-9c74-48c461f40bcc-kube-api-access-glgbw\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:16.010579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.010362    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d269e6-cdb0-4609-9c74-48c461f40bcc-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:16.650489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.650437    4274 generic.go:356] "Generic (PLEG): container finished" podID="14d269e6-cdb0-4609-9c74-48c461f40bcc" containerID="bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8" exitCode=0
Nov 03 21:05:16.666445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.650514    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"
Nov 03 21:05:16.666445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.650645    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" event={"ID":"14d269e6-cdb0-4609-9c74-48c461f40bcc","Type":"ContainerDied","Data":"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"}
Nov 03 21:05:16.666445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.650730    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b" event={"ID":"14d269e6-cdb0-4609-9c74-48c461f40bcc","Type":"ContainerDied","Data":"1e9b8f93f78818f6d63a318ed5af47e515357475215a02e74a2a1621212dc476"}
Nov 03 21:05:16.666445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.650760    4274 scope.go:122] "RemoveContainer" containerID="bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"
Nov 03 21:05:16.691580 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.691552    4274 scope.go:122] "RemoveContainer" containerID="bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"
Nov 03 21:05:16.692412 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:05:16.692387    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8\": container with ID starting with bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8 not found: ID does not exist" containerID="bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"
Nov 03 21:05:16.692518 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.692426    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8"} err="failed to get container status \"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8\": rpc error: code = NotFound desc = could not find container \"bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8\": container with ID starting with bb54d66be715b856bb55a5c35009fa1d85729b42e6bffa1f02196774472164a8 not found: ID does not exist"
Nov 03 21:05:16.822393 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.822349    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 21:05:16.825843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:16.825815    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-68s8b"]
Nov 03 21:05:17.230660 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:17.230602    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d269e6-cdb0-4609-9c74-48c461f40bcc" path="/var/lib/kubelet/pods/14d269e6-cdb0-4609-9c74-48c461f40bcc/volumes"
Nov 03 21:05:27.366694 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:05:27.366656    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:05:47.395053 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:47.395006    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 21:05:47.863368 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:47.863323    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy" containerID="cri-o://e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896" gracePeriod=30
Nov 03 21:05:53.134706 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.134652    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:05:53.284021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.283987    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284134    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284176    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284203    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284355    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284408    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284457    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:53.284551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284482    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:53.284551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284495    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284560    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284624    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\") pod \"f24e01f2-dffc-4769-aad9-43dde43e2094\" (UID: \"f24e01f2-dffc-4769-aad9-43dde43e2094\") "
Nov 03 21:05:53.284772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284718    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:53.284772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284731    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:53.284914 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.284809    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:53.285208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.285184    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.285324 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.285214    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.285324 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.285229    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24e01f2-dffc-4769-aad9-43dde43e2094-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.285324 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.285242    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.285324 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.285255    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.286351 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.286322    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "kube-api-access-qvcf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:53.286583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.286550    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:53.286728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.286709    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:53.286790 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.286714    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy" pod "f24e01f2-dffc-4769-aad9-43dde43e2094" (UID: "f24e01f2-dffc-4769-aad9-43dde43e2094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:53.388102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.386832    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcf2\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-kube-api-access-qvcf2\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.388102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.386892    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.388102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.387174    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.388102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.387209    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24e01f2-dffc-4769-aad9-43dde43e2094-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.903893 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.903858    4274 generic.go:356] "Generic (PLEG): container finished" podID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerID="e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896" exitCode=0
Nov 03 21:05:53.904103 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.903906    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"
Nov 03 21:05:53.904103 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.903915    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerDied","Data":"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"}
Nov 03 21:05:53.904103 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.903967    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8" event={"ID":"f24e01f2-dffc-4769-aad9-43dde43e2094","Type":"ContainerDied","Data":"e19190039c8861c2efb692187124a5ad78d5ae2f311c69c3505109074f8ac9e9"}
Nov 03 21:05:53.904103 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.903996    4274 scope.go:122] "RemoveContainer" containerID="e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"
Nov 03 21:05:53.914258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.914241    4274 scope.go:122] "RemoveContainer" containerID="b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"
Nov 03 21:05:53.922816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.922798    4274 scope.go:122] "RemoveContainer" containerID="e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"
Nov 03 21:05:53.923114 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:05:53.923093    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896\": container with ID starting with e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896 not found: ID does not exist" containerID="e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"
Nov 03 21:05:53.923182 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.923124    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896"} err="failed to get container status \"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896\": rpc error: code = NotFound desc = could not find container \"e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896\": container with ID starting with e57b34176c5b3fd2409f077af91d959f725ad3aed1527234e5290a1e7fffe896 not found: ID does not exist"
Nov 03 21:05:53.923182 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.923139    4274 scope.go:122] "RemoveContainer" containerID="b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"
Nov 03 21:05:53.923430 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:05:53.923412    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827\": container with ID starting with b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827 not found: ID does not exist" containerID="b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"
Nov 03 21:05:53.923478 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:53.923437    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827"} err="failed to get container status \"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827\": rpc error: code = NotFound desc = could not find container \"b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827\": container with ID starting with b0ddec310a49ec9137dd105cb0804dd70485158eb901b6451837aea8776fa827 not found: ID does not exist"
Nov 03 21:05:54.061257 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:54.061229    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 21:05:54.107131 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:54.107104    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2e3ea2-openshift-default-68f788d649-5s8n8"]
Nov 03 21:05:55.214134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:05:55.214097    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" path="/var/lib/kubelet/pods/f24e01f2-dffc-4769-aad9-43dde43e2094/volumes"
Nov 03 21:05:57.365816 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:05:57.365776    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:06:01.581970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:01.581938    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"]
Nov 03 21:06:01.627904 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:01.627869    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-37c5b0-cq588"]
Nov 03 21:06:01.630702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:01.630675    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:06:03.225104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:03.225061    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fdabddd-5498-4943-b50a-def73d96dfb3" path="/var/lib/kubelet/pods/4fdabddd-5498-4943-b50a-def73d96dfb3/volumes"
Nov 03 21:06:06.631899 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.631866    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" podUID="21d78142-b65b-405f-aaf8-0a5dd2e13114" containerName="payload-pre-processing" containerID="cri-o://3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450" gracePeriod=25
Nov 03 21:06:06.905797 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.905765    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:06:06.978489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.978454    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\" (UniqueName: \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\") pod \"21d78142-b65b-405f-aaf8-0a5dd2e13114\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") "
Nov 03 21:06:06.978626 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.978578    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\") pod \"21d78142-b65b-405f-aaf8-0a5dd2e13114\" (UID: \"21d78142-b65b-405f-aaf8-0a5dd2e13114\") "
Nov 03 21:06:06.978944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.978917    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume" pod "21d78142-b65b-405f-aaf8-0a5dd2e13114" (UID: "21d78142-b65b-405f-aaf8-0a5dd2e13114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:06.980487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.980463    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw" pod "21d78142-b65b-405f-aaf8-0a5dd2e13114" (UID: "21d78142-b65b-405f-aaf8-0a5dd2e13114"). InnerVolumeSpecName "kube-api-access-b4kgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:06.992246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.992213    4274 generic.go:356] "Generic (PLEG): container finished" podID="21d78142-b65b-405f-aaf8-0a5dd2e13114" containerID="3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450" exitCode=0
Nov 03 21:06:06.992442 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.992417    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" event={"ID":"21d78142-b65b-405f-aaf8-0a5dd2e13114","Type":"ContainerDied","Data":"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"}
Nov 03 21:06:06.992534 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.992517    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm" event={"ID":"21d78142-b65b-405f-aaf8-0a5dd2e13114","Type":"ContainerDied","Data":"f219e6f04d52a87d3ba59237a99b94a7b8b2328ee28fa414e2027a4976871da6"}
Nov 03 21:06:06.992590 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.992571    4274 scope.go:122] "RemoveContainer" containerID="3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"
Nov 03 21:06:06.992802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:06.992783    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"
Nov 03 21:06:07.008003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.007974    4274 scope.go:122] "RemoveContainer" containerID="3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"
Nov 03 21:06:07.008312 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:06:07.008290    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450\": container with ID starting with 3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450 not found: ID does not exist" containerID="3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"
Nov 03 21:06:07.008362 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.008320    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450"} err="failed to get container status \"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450\": rpc error: code = NotFound desc = could not find container \"3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450\": container with ID starting with 3aa3c882b59130429a1c8797bd359b815a82c51e98d672a57e8ffa477648b450 not found: ID does not exist"
Nov 03 21:06:07.080495 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.080461    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d78142-b65b-405f-aaf8-0a5dd2e13114-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:07.080495 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.080492    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4kgw\" (UniqueName: \"kubernetes.io/projected/21d78142-b65b-405f-aaf8-0a5dd2e13114-kube-api-access-b4kgw\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:07.120751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.120716    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:06:07.137649 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.137587    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d78b90-7b96559467-mvsfm"]
Nov 03 21:06:07.214045 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:07.214007    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d78142-b65b-405f-aaf8-0a5dd2e13114" path="/var/lib/kubelet/pods/21d78142-b65b-405f-aaf8-0a5dd2e13114/volumes"
Nov 03 21:06:27.366701 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:06:27.366665    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:06:51.098958 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:51.098926    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:06:51.099408 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:51.099166    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy" containerID="cri-o://b3a4dce4d203dd1617a37e25bc29fdf0a02141946ef7e514e771e3205fc7bc17" gracePeriod=30
Nov 03 21:06:56.316152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.316118    4274 generic.go:356] "Generic (PLEG): container finished" podID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerID="b3a4dce4d203dd1617a37e25bc29fdf0a02141946ef7e514e771e3205fc7bc17" exitCode=0
Nov 03 21:06:56.316492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.316187    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerDied","Data":"b3a4dce4d203dd1617a37e25bc29fdf0a02141946ef7e514e771e3205fc7bc17"}
Nov 03 21:06:56.316492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.316243    4274 scope.go:122] "RemoveContainer" containerID="ad4aff4849df5aad20fbd9057a734fbe4d81e4a7eb93aaccef4820fde2f575a9"
Nov 03 21:06:56.415434 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.415408    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:06:56.501125 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501091    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501161    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501207    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501272    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501347    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501389    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501424    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:56.501492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501446    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501515    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.501719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501543    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:56.501719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501582    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\") pod \"52cc125f-f1b1-4630-a220-4f5e7631ebac\" (UID: \"52cc125f-f1b1-4630-a220-4f5e7631ebac\") "
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.501958    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502011    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502132    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502230    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502243    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502259    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cc125f-f1b1-4630-a220-4f5e7631ebac-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502274    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.502313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.502288    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.503970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.503916    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:56.503970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.504204    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "kube-api-access-ph425". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:56.503970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.504381    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:06:56.504735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.504710    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy" pod "52cc125f-f1b1-4630-a220-4f5e7631ebac" (UID: "52cc125f-f1b1-4630-a220-4f5e7631ebac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:56.604123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.604086    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.604123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.604116    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph425\" (UniqueName: \"kubernetes.io/projected/52cc125f-f1b1-4630-a220-4f5e7631ebac-kube-api-access-ph425\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.604123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.604126    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.604418 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:56.604136    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cc125f-f1b1-4630-a220-4f5e7631ebac-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:57.323455 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:57.323419    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"
Nov 03 21:06:57.323889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:57.323464    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn" event={"ID":"52cc125f-f1b1-4630-a220-4f5e7631ebac","Type":"ContainerDied","Data":"0c63df998977d82a7be0ff1eff659b2fb6e74270b4487ff75f245ce23d0f738a"}
Nov 03 21:06:57.323889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:57.323524    4274 scope.go:122] "RemoveContainer" containerID="b3a4dce4d203dd1617a37e25bc29fdf0a02141946ef7e514e771e3205fc7bc17"
Nov 03 21:06:57.367140 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:06:57.367108    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:06:57.513944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:57.513914    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:06:57.558917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:57.558882    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-878695-openshift-default-84b87677f7-vm4qn"]
Nov 03 21:06:59.213846 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:06:59.213810    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" path="/var/lib/kubelet/pods/52cc125f-f1b1-4630-a220-4f5e7631ebac/volumes"
Nov 03 21:07:11.782218 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:11.782116    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:11.873060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:11.866119    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 21:07:12.296166 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.296126    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:07:12.296451 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.296419    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:12.396062 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.391118    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 21:07:12.454056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.445408    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:12.489311 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.489174    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:07:12.489311 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.489238    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:12.590286 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.590168    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:12.638387 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.638297    4274 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:12.661502 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:12.661156    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:19.528096 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.528065    4274 generic.go:356] "Generic (PLEG): container finished" podID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerID="ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922" exitCode=0
Nov 03 21:07:19.528542 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.528134    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerDied","Data":"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"}
Nov 03 21:07:19.528618 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.528597    4274 scope.go:122] "RemoveContainer" containerID="ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"
Nov 03 21:07:19.530008 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.529980    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:19.530119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.530096    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:19.530164 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:19.530133    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:20.542750 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:20.542702    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerStarted","Data":"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"}
Nov 03 21:07:21.437541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:21.437507    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:07:21.438486 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:21.438458    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:07:21.438621 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:21.438505    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:07:22.437367 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:22.437331    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:07:22.437778 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:22.437396    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:07:23.438103 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:23.438069    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:07:23.438516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:23.438128    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:07:24.437870 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:24.437832    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:07:24.438078 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:24.437897    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:07:24.583009 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:24.582979    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:07:25.437693 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.437656    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:07:25.437927 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.437725    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:07:25.592787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.592749    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:07:25.593551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593524    4274 cpu_manager.go:414] "containerMap: removing container" podUID="14d269e6-cdb0-4609-9c74-48c461f40bcc" containerName="maas-api"
Nov 03 21:07:25.593551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593551    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d269e6-cdb0-4609-9c74-48c461f40bcc" containerName="maas-api"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593574    4274 cpu_manager.go:414] "containerMap: removing container" podUID="21d78142-b65b-405f-aaf8-0a5dd2e13114" containerName="payload-pre-processing"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593583    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d78142-b65b-405f-aaf8-0a5dd2e13114" containerName="payload-pre-processing"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593621    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593630    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593671    4274 cpu_manager.go:414] "containerMap: removing container" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593679    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593689    4274 cpu_manager.go:414] "containerMap: removing container" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593713 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593696    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593819    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:25.593952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593833    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593843    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cc125f-f1b1-4630-a220-4f5e7631ebac" containerName="istio-proxy"
Nov 03 21:07:25.593952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593856    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d269e6-cdb0-4609-9c74-48c461f40bcc" containerName="maas-api"
Nov 03 21:07:25.593952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.593867    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d78142-b65b-405f-aaf8-0a5dd2e13114" containerName="payload-pre-processing"
Nov 03 21:07:25.597798 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.597779    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.669059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.663085    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e68f8112-openshift-default-dockercfg-cp47h\""
Nov 03 21:07:25.719595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.719503    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:07:25.795059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.791053    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795330 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795302    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpzv\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795483    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795517    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795679 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795647    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795753 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795731    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795834 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795816    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795906 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795855    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.795906 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.795883    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.896812 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896776    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896833    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpzv\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896890    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896915    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896949    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.896985    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897095    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897124    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897146    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897454    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897545 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897483    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897572    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897676 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897651    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.897935 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.897917    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.899763 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.899730    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.899926 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.899910    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.954722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.954687    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:25.955315 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:25.955281    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpzv\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:26.210611 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.210583    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:26.441642 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.441612    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:07:26.442604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.442587    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:07:26.679503 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.679470    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:07:26.683189 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:07:26.683164    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa51e164_ffd2_48f0_96b4_ae93155284d6.slice/crio-7ad4b7417f999dbe62c7c529605da30c52655d87910b4645eafbf2db2a16b590 WatchSource:0}: Error finding container 7ad4b7417f999dbe62c7c529605da30c52655d87910b4645eafbf2db2a16b590: Status 404 returned error can't find the container with id 7ad4b7417f999dbe62c7c529605da30c52655d87910b4645eafbf2db2a16b590
Nov 03 21:07:26.685850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.685814    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:26.685971 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.685898    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:26.685971 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:26.685946    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:27.396578 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:07:27.396545    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:07:27.632555 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:27.632514    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" event={"ID":"aa51e164-ffd2-48f0-96b4-ae93155284d6","Type":"ContainerStarted","Data":"20fc367b8e98be51974e4994ef65eb8540ee5c53a91b0327dc45035fb2c0fa4f"}
Nov 03 21:07:27.632743 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:27.632572    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" event={"ID":"aa51e164-ffd2-48f0-96b4-ae93155284d6","Type":"ContainerStarted","Data":"7ad4b7417f999dbe62c7c529605da30c52655d87910b4645eafbf2db2a16b590"}
Nov 03 21:07:27.765066 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:27.761887    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podStartSLOduration=2.761859306 podStartE2EDuration="2.761859306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:27.745162568 +0000 UTC m=+2272.671732810" watchObservedRunningTime="2026-11-03 21:07:27.761859306 +0000 UTC m=+2272.688429544"
Nov 03 21:07:28.211237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:28.211156    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:28.212234 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:28.212213    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:28.212304 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:28.212285    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 21:07:29.211680 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:29.211646    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:29.212168 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:29.211712    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 21:07:30.211636 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:30.211604    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:30.211823 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:30.211656    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 21:07:31.211899 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:31.211862    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:31.212276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:31.211921    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 21:07:32.211751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:32.211714    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:32.212110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:32.212073    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 21:07:33.271068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:33.271005    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:33.271603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:33.271486    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:33.276874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:33.276850    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:07:34.570250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:34.570212    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:07:34.570698 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:34.570425    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy" containerID="cri-o://cf5079141c9a8e12102094d13d121d79e5b87d696ae3c32d03f5ecd74d227f5d" gracePeriod=30
Nov 03 21:07:39.756405 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.756358    4274 generic.go:356] "Generic (PLEG): container finished" podID="629b1544-5969-4684-903c-8923c198ea8f" containerID="cf5079141c9a8e12102094d13d121d79e5b87d696ae3c32d03f5ecd74d227f5d" exitCode=0
Nov 03 21:07:39.756837 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.756517    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerDied","Data":"cf5079141c9a8e12102094d13d121d79e5b87d696ae3c32d03f5ecd74d227f5d"}
Nov 03 21:07:39.756837 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.756577    4274 scope.go:122] "RemoveContainer" containerID="7e1285f376d1e27d3f36b8fb2d36a801931382d9d988eea3129703bc0855279c"
Nov 03 21:07:39.842471 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.842410    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:07:39.947275 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947241    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947469 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947289    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947469 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947386    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947471    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947514    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947527    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:39.947735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947595    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947630    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947667    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947876    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\") pod \"629b1544-5969-4684-903c-8923c198ea8f\" (UID: \"629b1544-5969-4684-903c-8923c198ea8f\") "
Nov 03 21:07:39.947944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947879    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:39.947944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.947890    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:39.948164 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948117    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:39.948259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948241    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:39.948686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948666    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629b1544-5969-4684-903c-8923c198ea8f-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:39.948727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948694    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:39.948727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948709    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:39.948792 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948723    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:39.948792 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.948739    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:39.949726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.949694    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:39.949726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.949712    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "kube-api-access-c6q6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:39.949933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.949911    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:07:39.950006 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:39.949969    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy" pod "629b1544-5969-4684-903c-8923c198ea8f" (UID: "629b1544-5969-4684-903c-8923c198ea8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:40.050536 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.050500    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.050536 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.050530    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629b1544-5969-4684-903c-8923c198ea8f-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.050726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.050562    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6q6s\" (UniqueName: \"kubernetes.io/projected/629b1544-5969-4684-903c-8923c198ea8f-kube-api-access-c6q6s\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.050726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.050577    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629b1544-5969-4684-903c-8923c198ea8f-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.719200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.719169    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:07:40.719882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.719856    4274 cpu_manager.go:414] "containerMap: removing container" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:40.719882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.719880    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:40.720121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.719936    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:40.720121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.719942    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:40.720121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.720078    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24e01f2-dffc-4769-aad9-43dde43e2094" containerName="istio-proxy"
Nov 03 21:07:40.720121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.720088    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:40.720121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.720121    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:40.723739 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.723722    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.763933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.763909    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"
Nov 03 21:07:40.764317 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.763957    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd" event={"ID":"629b1544-5969-4684-903c-8923c198ea8f","Type":"ContainerDied","Data":"77b1c711c63873a7f6e435194afbffdac7938d41ebe41cf592ad20b561d2cd9c"}
Nov 03 21:07:40.764317 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.764014    4274 scope.go:122] "RemoveContainer" containerID="cf5079141c9a8e12102094d13d121d79e5b87d696ae3c32d03f5ecd74d227f5d"
Nov 03 21:07:40.862395 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862366    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4q4q\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862420    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862523    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862578    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862622    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862756 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862691    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862756 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862740    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862837 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862826    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.862874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.862859    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.932391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.932357    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:07:40.946366 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.946336    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:07:40.951306 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.951280    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8448c9-openshift-default-54b7d69b79-c4sqd"]
Nov 03 21:07:40.963709 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963681    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.963806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963730    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.963806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963778    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4q4q\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.963931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963820    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.963931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963871    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.963931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963926    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.963956    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964029    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964202 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964119    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964202 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964176    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964306 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964275    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964528 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964459    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964530    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.964995 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.964970    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.966427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.966403    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:40.966801 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:40.966781    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:41.020748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.020726    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:41.020878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.020859    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4q4q\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:41.037180 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.037162    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:41.213624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.213594    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629b1544-5969-4684-903c-8923c198ea8f" path="/var/lib/kubelet/pods/629b1544-5969-4684-903c-8923c198ea8f/volumes"
Nov 03 21:07:41.467757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.467717    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:07:41.468162 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:07:41.468131    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78b7483_e549_445c_af9e_9b85fdc8f77f.slice/crio-a5171fb2d5d2d225e006549c0881ca258c034239345db3f20cf00a35f2c295f9 WatchSource:0}: Error finding container a5171fb2d5d2d225e006549c0881ca258c034239345db3f20cf00a35f2c295f9: Status 404 returned error can't find the container with id a5171fb2d5d2d225e006549c0881ca258c034239345db3f20cf00a35f2c295f9
Nov 03 21:07:41.470974 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.470939    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:41.471121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.471029    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:41.471121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.471104    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:41.772031 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.771999    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" event={"ID":"e78b7483-e549-445c-af9e-9b85fdc8f77f","Type":"ContainerStarted","Data":"4e8cddf5d55e12f212b6aa9f08202e1450953b53c434279591fba7422eac053b"}
Nov 03 21:07:41.772450 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.772071    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" event={"ID":"e78b7483-e549-445c-af9e-9b85fdc8f77f","Type":"ContainerStarted","Data":"a5171fb2d5d2d225e006549c0881ca258c034239345db3f20cf00a35f2c295f9"}
Nov 03 21:07:41.887930 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:41.887865    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podStartSLOduration=1.8878494099999998 podStartE2EDuration="1.88784941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:41.885117355 +0000 UTC m=+2286.811687593" watchObservedRunningTime="2026-11-03 21:07:41.88784941 +0000 UTC m=+2286.814419649"
Nov 03 21:07:42.038101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:42.037990    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:42.039407 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:42.039384    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:42.039473 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:42.039435    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:43.037714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:43.037680    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:43.038244 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:43.037737    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:44.038259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:44.038218    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:44.038689 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:44.038295    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:45.038150 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:45.038069    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:45.038150 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:45.038135    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:46.038401 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:46.038366    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:46.038801 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:46.038429    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:47.038053 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:47.037998    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:47.038236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:47.038086    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:48.037574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:48.037539    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 21:07:48.037947 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:48.037594    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 21:07:49.042404 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:49.042374    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:49.042852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:49.042754    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:49.043532 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:49.043511    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:07:55.841280 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.841249    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:07:55.841781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.841767    4274 cpu_manager.go:414] "containerMap: removing container" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:55.841829 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.841782    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629b1544-5969-4684-903c-8923c198ea8f" containerName="istio-proxy"
Nov 03 21:07:55.846458 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.846440    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996534 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996495    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9b4\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996534 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996534    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996743 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996609    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996794 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996738    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996831 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996790    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996896 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996871    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.996964 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.996932    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.997019 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.997003    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:55.997115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:55.997099    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.040846 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.040802    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:07:56.098889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.098812    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.098897    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.098945    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.098990    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099229 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099064    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099229 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099124    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099229 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099171    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099272    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099310    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099320    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9b4\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099361    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099560 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099381    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099617 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099576    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.099865 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.099843    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.101440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.101412    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.101661 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.101644    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.156383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.156364    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.156475 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.156453    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9b4\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.162367 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.162350    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:56.579452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.579425    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:07:56.580769 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:07:56.580740    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac63bf4_034e_4643_9375_c2e01f29f143.slice/crio-39c0c48d419221f449b28e9521a6656200b19106a9a5ae29fce1d7490f3f9b9a WatchSource:0}: Error finding container 39c0c48d419221f449b28e9521a6656200b19106a9a5ae29fce1d7490f3f9b9a: Status 404 returned error can't find the container with id 39c0c48d419221f449b28e9521a6656200b19106a9a5ae29fce1d7490f3f9b9a
Nov 03 21:07:56.583690 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.583659    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:56.583779 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.583741    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:56.583820 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.583777    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:56.874010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.873918    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" event={"ID":"4ac63bf4-034e-4643-9375-c2e01f29f143","Type":"ContainerStarted","Data":"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"}
Nov 03 21:07:56.874010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.873973    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" event={"ID":"4ac63bf4-034e-4643-9375-c2e01f29f143","Type":"ContainerStarted","Data":"39c0c48d419221f449b28e9521a6656200b19106a9a5ae29fce1d7490f3f9b9a"}
Nov 03 21:07:56.989173 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:56.989121    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podStartSLOduration=1.9891052569999998 podStartE2EDuration="1.989105257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:56.986996774 +0000 UTC m=+2301.913567014" watchObservedRunningTime="2026-11-03 21:07:56.989105257 +0000 UTC m=+2301.915675493"
Nov 03 21:07:57.170115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:57.166761    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:07:57.170115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:57.166837    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:57.170115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:57.166890    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:07:57.394386 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:07:57.394348    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:07:58.170425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:58.170388    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:58.170830 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:58.170449    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:07:59.169061 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:59.168965    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:59.169061 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:07:59.169027    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:08:00.168861 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:00.168825    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:08:00.169462 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:00.169431    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:08:01.166175 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:01.166125    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:08:01.166916 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:01.166882    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:08:02.168059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:02.162990    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:08:02.170175 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:02.170127    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 03 21:08:03.236249 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:03.236202    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:08:03.238144 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:03.238105    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:08:03.241303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:03.241277    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:08:15.684280 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:15.684234    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 21:08:15.739679 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:15.739647    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 21:08:27.366192 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:08:27.366155    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:08:30.710275 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:30.710214    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main" containerID="cri-o://fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed" gracePeriod=45
Nov 03 21:08:30.760284 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:30.760241    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main" containerID="cri-o://5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d" gracePeriod=45
Nov 03 21:08:31.042791 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.042765    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 21:08:31.045886 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.045865    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 21:08:31.115980 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.115945    4274 generic.go:356] "Generic (PLEG): container finished" podID="630ef936-48ad-44bc-aa99-e35d2f503099" containerID="fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed" exitCode=0
Nov 03 21:08:31.116196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.115998    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"
Nov 03 21:08:31.116196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.116021    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" event={"ID":"630ef936-48ad-44bc-aa99-e35d2f503099","Type":"ContainerDied","Data":"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"}
Nov 03 21:08:31.116196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.116089    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4" event={"ID":"630ef936-48ad-44bc-aa99-e35d2f503099","Type":"ContainerDied","Data":"65f3d5f7f8114234e0c2363e874f29a012923fbdcf5177a74716be015cc807a4"}
Nov 03 21:08:31.116196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.116112    4274 scope.go:122] "RemoveContainer" containerID="fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"
Nov 03 21:08:31.117400 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.117376    4274 generic.go:356] "Generic (PLEG): container finished" podID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerID="5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d" exitCode=0
Nov 03 21:08:31.117516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.117413    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"
Nov 03 21:08:31.117516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.117424    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" event={"ID":"9734fd48-ac42-436e-8e0a-9e8b3895ddc5","Type":"ContainerDied","Data":"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"}
Nov 03 21:08:31.117516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.117452    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz" event={"ID":"9734fd48-ac42-436e-8e0a-9e8b3895ddc5","Type":"ContainerDied","Data":"c1a94fa8afd624edfba94a22c637b406106426e299a09f7ec88a74b5f1c56df5"}
Nov 03 21:08:31.125418 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.125397    4274 scope.go:122] "RemoveContainer" containerID="fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"
Nov 03 21:08:31.125678 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:08:31.125656    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed\": container with ID starting with fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed not found: ID does not exist" containerID="fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"
Nov 03 21:08:31.125757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.125690    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed"} err="failed to get container status \"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed\": rpc error: code = NotFound desc = could not find container \"fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed\": container with ID starting with fb530bf595f4a38b3c1890ecd112a84729c22000bd240e5945c25503ce853aed not found: ID does not exist"
Nov 03 21:08:31.125757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.125713    4274 scope.go:122] "RemoveContainer" containerID="5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"
Nov 03 21:08:31.134190 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.134170    4274 scope.go:122] "RemoveContainer" containerID="5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"
Nov 03 21:08:31.134486 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:08:31.134468    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d\": container with ID starting with 5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d not found: ID does not exist" containerID="5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"
Nov 03 21:08:31.134551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.134494    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d"} err="failed to get container status \"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d\": rpc error: code = NotFound desc = could not find container \"5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d\": container with ID starting with 5dc9c50205254a4290b32a7408575dbd4f5f5f7bbade905eaeda47d7c5e2df8d not found: ID does not exist"
Nov 03 21:08:31.159674 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.159300    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.159898 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.159813    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.159898 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.159859    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\" (UniqueName: \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.159946    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\" (UniqueName: \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.159986    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160062    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160134    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160195    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\" (UniqueName: \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.160241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160237    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.160557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160297    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\") pod \"630ef936-48ad-44bc-aa99-e35d2f503099\" (UID: \"630ef936-48ad-44bc-aa99-e35d2f503099\") "
Nov 03 21:08:31.160557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160339    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.160557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.160402    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\") pod \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\" (UID: \"9734fd48-ac42-436e-8e0a-9e8b3895ddc5\") "
Nov 03 21:08:31.161723 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.161251    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.161723 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.161323    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.161723 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.161464    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.161723 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.161644    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.162109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.161988    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.162382 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.162361    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.164028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.163978    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "kube-api-access-rrn6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:31.164028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.164076    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:31.166782 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.166711    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.166782 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.166716    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm" pod "630ef936-48ad-44bc-aa99-e35d2f503099" (UID: "630ef936-48ad-44bc-aa99-e35d2f503099"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.166782 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.166750    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "kube-api-access-hw4r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:31.166984 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.166964    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs" pod "9734fd48-ac42-436e-8e0a-9e8b3895ddc5" (UID: "9734fd48-ac42-436e-8e0a-9e8b3895ddc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262371    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262407    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262421    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262434    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262447    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262460    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262474    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4r5\" (UniqueName: \"kubernetes.io/projected/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-kube-api-access-hw4r5\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262487    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrn6t\" (UniqueName: \"kubernetes.io/projected/630ef936-48ad-44bc-aa99-e35d2f503099-kube-api-access-rrn6t\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262500    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9734fd48-ac42-436e-8e0a-9e8b3895ddc5-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262514    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262527    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630ef936-48ad-44bc-aa99-e35d2f503099-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.262734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.262539    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630ef936-48ad-44bc-aa99-e35d2f503099-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.525648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.525521    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 21:08:31.529306 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.529282    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/simulator-w4-3d2008-kserve-679447b6fb-c6jlz"]
Nov 03 21:08:31.632364 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.632334    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 21:08:31.637569 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:31.637548    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-w4-3d2008-kserve-78b6b48fd4-2xzf4"]
Nov 03 21:08:33.214762 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:33.214731    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" path="/var/lib/kubelet/pods/630ef936-48ad-44bc-aa99-e35d2f503099/volumes"
Nov 03 21:08:33.215210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:08:33.215194    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" path="/var/lib/kubelet/pods/9734fd48-ac42-436e-8e0a-9e8b3895ddc5/volumes"
Nov 03 21:08:57.371566 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:08:57.371525    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:09:27.365298 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:09:27.365263    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:09:27.611691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.611648    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:09:27.611938 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.611896    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" podUID="b0215b97-42ea-404d-9afa-fc82cbbf0740" containerName="maas-api" containerID="cri-o://788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82" gracePeriod=30
Nov 03 21:09:27.917062 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.916999    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:09:27.965612 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.965587    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") pod \"b0215b97-42ea-404d-9afa-fc82cbbf0740\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") "
Nov 03 21:09:27.965737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.965662    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\") pod \"b0215b97-42ea-404d-9afa-fc82cbbf0740\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") "
Nov 03 21:09:27.965870 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.965852    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\") pod \"b0215b97-42ea-404d-9afa-fc82cbbf0740\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") "
Nov 03 21:09:27.966069 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.966025    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\" (UniqueName: \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\") pod \"b0215b97-42ea-404d-9afa-fc82cbbf0740\" (UID: \"b0215b97-42ea-404d-9afa-fc82cbbf0740\") "
Nov 03 21:09:27.966285 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.966260    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca" pod "b0215b97-42ea-404d-9afa-fc82cbbf0740" (UID: "b0215b97-42ea-404d-9afa-fc82cbbf0740"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:27.966669 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.966647    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0215b97-42ea-404d-9afa-fc82cbbf0740-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:27.967755 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.967737    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs" pod "b0215b97-42ea-404d-9afa-fc82cbbf0740" (UID: "b0215b97-42ea-404d-9afa-fc82cbbf0740"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:27.967810 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.967762    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls" pod "b0215b97-42ea-404d-9afa-fc82cbbf0740" (UID: "b0215b97-42ea-404d-9afa-fc82cbbf0740"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:27.968080 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:27.968058    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp" pod "b0215b97-42ea-404d-9afa-fc82cbbf0740" (UID: "b0215b97-42ea-404d-9afa-fc82cbbf0740"). InnerVolumeSpecName "kube-api-access-xhxrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:28.068384 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.068347    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:28.068384 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.068388    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxrp\" (UniqueName: \"kubernetes.io/projected/b0215b97-42ea-404d-9afa-fc82cbbf0740-kube-api-access-xhxrp\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:28.068610 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.068404    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0215b97-42ea-404d-9afa-fc82cbbf0740-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:28.522310 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.522272    4274 generic.go:356] "Generic (PLEG): container finished" podID="b0215b97-42ea-404d-9afa-fc82cbbf0740" containerID="788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82" exitCode=0
Nov 03 21:09:28.522838 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.522323    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"
Nov 03 21:09:28.522838 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.522318    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" event={"ID":"b0215b97-42ea-404d-9afa-fc82cbbf0740","Type":"ContainerDied","Data":"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"}
Nov 03 21:09:28.522838 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.522457    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4" event={"ID":"b0215b97-42ea-404d-9afa-fc82cbbf0740","Type":"ContainerDied","Data":"064b1a196cf7e4ea6534fa3236358a52dd0ea4beac0e3fb7cd345eef5e9ea07d"}
Nov 03 21:09:28.522838 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.522489    4274 scope.go:122] "RemoveContainer" containerID="788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"
Nov 03 21:09:28.533120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.533103    4274 scope.go:122] "RemoveContainer" containerID="788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"
Nov 03 21:09:28.533385 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:09:28.533349    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82\": container with ID starting with 788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82 not found: ID does not exist" containerID="788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"
Nov 03 21:09:28.533496 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.533390    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82"} err="failed to get container status \"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82\": rpc error: code = NotFound desc = could not find container \"788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82\": container with ID starting with 788791efd3ce3189651af8bebba5eef69b4728d91b694d41810b609bb4ca1f82 not found: ID does not exist"
Nov 03 21:09:28.707226 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.707189    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:09:28.710345 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:28.710313    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-9ce672-56945f5898-tmjp4"]
Nov 03 21:09:29.215358 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:29.215326    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0215b97-42ea-404d-9afa-fc82cbbf0740" path="/var/lib/kubelet/pods/b0215b97-42ea-404d-9afa-fc82cbbf0740/volumes"
Nov 03 21:09:42.759169 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:42.759135    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 21:09:45.289829 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.289793    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 21:09:45.290247 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.290015    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" podUID="dc238b02-aaa7-418b-990b-ed194b40c725" containerName="maas-api" containerID="cri-o://c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba" gracePeriod=30
Nov 03 21:09:45.594914 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.594890    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 21:09:45.632721 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.632687    4274 generic.go:356] "Generic (PLEG): container finished" podID="dc238b02-aaa7-418b-990b-ed194b40c725" containerID="c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba" exitCode=0
Nov 03 21:09:45.632852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.632733    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"
Nov 03 21:09:45.632852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.632764    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" event={"ID":"dc238b02-aaa7-418b-990b-ed194b40c725","Type":"ContainerDied","Data":"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"}
Nov 03 21:09:45.632852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.632813    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj" event={"ID":"dc238b02-aaa7-418b-990b-ed194b40c725","Type":"ContainerDied","Data":"7412e12ec2ce8fc8265399634b77121d1f352e68e1c8c8a61151f6be6b1af546"}
Nov 03 21:09:45.632852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.632842    4274 scope.go:122] "RemoveContainer" containerID="c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"
Nov 03 21:09:45.642152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.642133    4274 scope.go:122] "RemoveContainer" containerID="c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"
Nov 03 21:09:45.642405 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:09:45.642383    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba\": container with ID starting with c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba not found: ID does not exist" containerID="c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"
Nov 03 21:09:45.642472 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.642412    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba"} err="failed to get container status \"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba\": rpc error: code = NotFound desc = could not find container \"c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba\": container with ID starting with c96984f8967c78c538bb8f5cd25fa55e016d5cee1a9e076f784a52817bc657ba not found: ID does not exist"
Nov 03 21:09:45.733858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.733823    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") pod \"dc238b02-aaa7-418b-990b-ed194b40c725\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") "
Nov 03 21:09:45.734003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.733945    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\") pod \"dc238b02-aaa7-418b-990b-ed194b40c725\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") "
Nov 03 21:09:45.734089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.734048    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\" (UniqueName: \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\") pod \"dc238b02-aaa7-418b-990b-ed194b40c725\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") "
Nov 03 21:09:45.734188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.734164    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\") pod \"dc238b02-aaa7-418b-990b-ed194b40c725\" (UID: \"dc238b02-aaa7-418b-990b-ed194b40c725\") "
Nov 03 21:09:45.734376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.734348    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca" pod "dc238b02-aaa7-418b-990b-ed194b40c725" (UID: "dc238b02-aaa7-418b-990b-ed194b40c725"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:45.734693 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.734673    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc238b02-aaa7-418b-990b-ed194b40c725-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:45.736678 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.736543    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls" pod "dc238b02-aaa7-418b-990b-ed194b40c725" (UID: "dc238b02-aaa7-418b-990b-ed194b40c725"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:45.736678 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.736641    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs" pod "dc238b02-aaa7-418b-990b-ed194b40c725" (UID: "dc238b02-aaa7-418b-990b-ed194b40c725"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:45.736678 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.736652    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx" pod "dc238b02-aaa7-418b-990b-ed194b40c725" (UID: "dc238b02-aaa7-418b-990b-ed194b40c725"). InnerVolumeSpecName "kube-api-access-g2gkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:45.836268 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.836190    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:45.836268 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.836218    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc238b02-aaa7-418b-990b-ed194b40c725-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:45.836268 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:45.836229    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2gkx\" (UniqueName: \"kubernetes.io/projected/dc238b02-aaa7-418b-990b-ed194b40c725-kube-api-access-g2gkx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:46.047698 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:46.047656    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 21:09:46.054176 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:46.054151    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3d2008-6b9d566fbd-vdwpj"]
Nov 03 21:09:47.213459 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:47.213428    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc238b02-aaa7-418b-990b-ed194b40c725" path="/var/lib/kubelet/pods/dc238b02-aaa7-418b-990b-ed194b40c725/volumes"
Nov 03 21:09:57.368986 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:09:57.368943    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:09:57.782194 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:57.782141    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main" containerID="cri-o://836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d" gracePeriod=45
Nov 03 21:09:58.055935 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.055908    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 21:09:58.123580 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123553    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\" (UniqueName: \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.123715 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123666    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.123801 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123789    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\" (UniqueName: \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.123860 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123808    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.123860 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123854    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.123958 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.123918    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\") pod \"feff67ee-3424-465b-935f-284debb0350a\" (UID: \"feff67ee-3424-465b-935f-284debb0350a\") "
Nov 03 21:09:58.124133 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124109    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.124232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124168    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.124290 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124267    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.124560 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124525    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.124560 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124551    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.124704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.124566    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.125946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.125922    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:58.126089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.126062    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.126157 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.126086    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj" pod "feff67ee-3424-465b-935f-284debb0350a" (UID: "feff67ee-3424-465b-935f-284debb0350a"). InnerVolumeSpecName "kube-api-access-gp4vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:58.226592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.226547    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4vj\" (UniqueName: \"kubernetes.io/projected/feff67ee-3424-465b-935f-284debb0350a-kube-api-access-gp4vj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.226592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.226596    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feff67ee-3424-465b-935f-284debb0350a-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.226895 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.226612    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feff67ee-3424-465b-935f-284debb0350a-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.454600 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.454514    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:09:58.456051 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.454804    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" containerID="cri-o://20fc367b8e98be51974e4994ef65eb8540ee5c53a91b0327dc45035fb2c0fa4f" gracePeriod=30
Nov 03 21:09:58.462111 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.462085    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:09:58.462363 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.462317    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" containerID="cri-o://083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b" gracePeriod=30
Nov 03 21:09:58.473672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.473644    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:09:58.473872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.473849    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy" containerID="cri-o://4e8cddf5d55e12f212b6aa9f08202e1450953b53c434279591fba7422eac053b" gracePeriod=30
Nov 03 21:09:58.729094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.728989    4274 generic.go:356] "Generic (PLEG): container finished" podID="feff67ee-3424-465b-935f-284debb0350a" containerID="836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d" exitCode=0
Nov 03 21:09:58.729094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.729047    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"
Nov 03 21:09:58.729094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.729078    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" event={"ID":"feff67ee-3424-465b-935f-284debb0350a","Type":"ContainerDied","Data":"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"}
Nov 03 21:09:58.729421 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.729121    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd" event={"ID":"feff67ee-3424-465b-935f-284debb0350a","Type":"ContainerDied","Data":"703795a601d151efa9ef4dde4f692d31c4dc51e9fadc8b3d631b0fff8317edbf"}
Nov 03 21:09:58.729421 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.729140    4274 scope.go:122] "RemoveContainer" containerID="836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"
Nov 03 21:09:58.739635 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.739618    4274 scope.go:122] "RemoveContainer" containerID="836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"
Nov 03 21:09:58.739956 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:09:58.739925    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d\": container with ID starting with 836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d not found: ID does not exist" containerID="836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"
Nov 03 21:09:58.740023 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.739969    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d"} err="failed to get container status \"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d\": rpc error: code = NotFound desc = could not find container \"836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d\": container with ID starting with 836dd0ec09b28308a13ea70e3d238c43789e18d98bf5f05f4b5681f396b9a72d not found: ID does not exist"
Nov 03 21:09:58.930100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.930070    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 21:09:58.934659 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:58.934631    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-w5-2a5ec9-kserve-7cd54f88f5-npbfd"]
Nov 03 21:09:59.214163 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:09:59.214134    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feff67ee-3424-465b-935f-284debb0350a" path="/var/lib/kubelet/pods/feff67ee-3424-465b-935f-284debb0350a/volumes"
Nov 03 21:10:03.240387 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.240348    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:03.240777 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.240409    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:03.273617 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.273585    4274 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:03.273772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.273636    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:03.772968 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.772929    4274 generic.go:356] "Generic (PLEG): container finished" podID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerID="20fc367b8e98be51974e4994ef65eb8540ee5c53a91b0327dc45035fb2c0fa4f" exitCode=0
Nov 03 21:10:03.773175 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.773065    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" event={"ID":"aa51e164-ffd2-48f0-96b4-ae93155284d6","Type":"ContainerDied","Data":"20fc367b8e98be51974e4994ef65eb8540ee5c53a91b0327dc45035fb2c0fa4f"}
Nov 03 21:10:03.775323 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.775294    4274 generic.go:356] "Generic (PLEG): container finished" podID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerID="4e8cddf5d55e12f212b6aa9f08202e1450953b53c434279591fba7422eac053b" exitCode=0
Nov 03 21:10:03.775467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.775335    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" event={"ID":"e78b7483-e549-445c-af9e-9b85fdc8f77f","Type":"ContainerDied","Data":"4e8cddf5d55e12f212b6aa9f08202e1450953b53c434279591fba7422eac053b"}
Nov 03 21:10:03.776147 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.776124    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:10:03.777155 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.777131    4274 generic.go:356] "Generic (PLEG): container finished" podID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerID="083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b" exitCode=0
Nov 03 21:10:03.777264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.777155    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" event={"ID":"4ac63bf4-034e-4643-9375-c2e01f29f143","Type":"ContainerDied","Data":"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"}
Nov 03 21:10:03.777264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.777219    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq" event={"ID":"4ac63bf4-034e-4643-9375-c2e01f29f143","Type":"ContainerDied","Data":"39c0c48d419221f449b28e9521a6656200b19106a9a5ae29fce1d7490f3f9b9a"}
Nov 03 21:10:03.777264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.777253    4274 scope.go:122] "RemoveContainer" containerID="083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"
Nov 03 21:10:03.791776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.791754    4274 scope.go:122] "RemoveContainer" containerID="083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"
Nov 03 21:10:03.792120 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:10:03.792091    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b\": container with ID starting with 083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b not found: ID does not exist" containerID="083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"
Nov 03 21:10:03.792198 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.792132    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b"} err="failed to get container status \"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b\": rpc error: code = NotFound desc = could not find container \"083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b\": container with ID starting with 083701e88d76af32a5c3281f43e49fde966b545245180dc2f0040163ff97e94b not found: ID does not exist"
Nov 03 21:10:03.792909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.792889    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:10:03.818410 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.818390    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:10:03.928455 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928417    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928465    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928489    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928518    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928540    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928558    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.928648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928611    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.928944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928754    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.928944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928811    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.928944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928816    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.928944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928816    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.928944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.928885    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929061    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929095    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929107    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929188    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929211    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.929238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929229    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929259    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929300    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929316    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929356    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929401    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929477    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929510    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929553    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929589    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929639    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929673    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\") pod \"e78b7483-e549-445c-af9e-9b85fdc8f77f\" (UID: \"e78b7483-e549-445c-af9e-9b85fdc8f77f\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929707    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929735    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929766    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929798    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\") pod \"4ac63bf4-034e-4643-9375-c2e01f29f143\" (UID: \"4ac63bf4-034e-4643-9375-c2e01f29f143\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929840    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\") pod \"aa51e164-ffd2-48f0-96b4-ae93155284d6\" (UID: \"aa51e164-ffd2-48f0-96b4-ae93155284d6\") "
Nov 03 21:10:03.929970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.929884    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.930456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930166    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.930456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930399    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.930558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930490    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.930689 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930666    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.930751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930681    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.930931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930908    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.931026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930963    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.930983    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931000    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa51e164-ffd2-48f0-96b4-ae93155284d6-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931014    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931028    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931063    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931078    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931090    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931103    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931117    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931130    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931143    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78b7483-e549-445c-af9e-9b85fdc8f77f-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931155    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.931749 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931368    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.931749 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.931380    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.932364 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.932336    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "kube-api-access-t4q4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.932605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.932583    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:03.933603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933574    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.933702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933577    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "kube-api-access-6kpzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.933702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933583    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:03.933702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933631    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "kube-api-access-9n9b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.933922 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933834    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.933922 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.933852    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.934098 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.934075    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token" pod "e78b7483-e549-445c-af9e-9b85fdc8f77f" (UID: "e78b7483-e549-445c-af9e-9b85fdc8f77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.934466 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.934444    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy" pod "aa51e164-ffd2-48f0-96b4-ae93155284d6" (UID: "aa51e164-ffd2-48f0-96b4-ae93155284d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.934466 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:03.934456    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo" pod "4ac63bf4-034e-4643-9375-c2e01f29f143" (UID: "4ac63bf4-034e-4643-9375-c2e01f29f143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032130    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032165    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac63bf4-034e-4643-9375-c2e01f29f143-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032175    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032185    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032195    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032204    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032214    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032222    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032230    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78b7483-e549-445c-af9e-9b85fdc8f77f-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032240    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4q4q\" (UniqueName: \"kubernetes.io/projected/e78b7483-e549-445c-af9e-9b85fdc8f77f-kube-api-access-t4q4q\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032250    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kpzv\" (UniqueName: \"kubernetes.io/projected/aa51e164-ffd2-48f0-96b4-ae93155284d6-kube-api-access-6kpzv\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032258    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa51e164-ffd2-48f0-96b4-ae93155284d6-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032268    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac63bf4-034e-4643-9375-c2e01f29f143-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.032521 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.032276    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n9b4\" (UniqueName: \"kubernetes.io/projected/4ac63bf4-034e-4643-9375-c2e01f29f143-kube-api-access-9n9b4\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.785192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.785162    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"
Nov 03 21:10:04.785641 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.785199    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454" event={"ID":"aa51e164-ffd2-48f0-96b4-ae93155284d6","Type":"ContainerDied","Data":"7ad4b7417f999dbe62c7c529605da30c52655d87910b4645eafbf2db2a16b590"}
Nov 03 21:10:04.785641 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.785273    4274 scope.go:122] "RemoveContainer" containerID="20fc367b8e98be51974e4994ef65eb8540ee5c53a91b0327dc45035fb2c0fa4f"
Nov 03 21:10:04.786786 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.786760    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"
Nov 03 21:10:04.786909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.786777    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz" event={"ID":"e78b7483-e549-445c-af9e-9b85fdc8f77f","Type":"ContainerDied","Data":"a5171fb2d5d2d225e006549c0881ca258c034239345db3f20cf00a35f2c295f9"}
Nov 03 21:10:04.788056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.788003    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"
Nov 03 21:10:04.797804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.797789    4274 scope.go:122] "RemoveContainer" containerID="4e8cddf5d55e12f212b6aa9f08202e1450953b53c434279591fba7422eac053b"
Nov 03 21:10:04.940025 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.939995    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:10:04.986020 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:04.985986    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-rwljq"]
Nov 03 21:10:05.053717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.053620    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:10:05.056389 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.056364    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-9f454"]
Nov 03 21:10:05.175287 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.175253    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:10:05.203430 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.203406    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-hgzdz"]
Nov 03 21:10:05.214350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.214314    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" path="/var/lib/kubelet/pods/4ac63bf4-034e-4643-9375-c2e01f29f143/volumes"
Nov 03 21:10:05.214890 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.214867    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" path="/var/lib/kubelet/pods/aa51e164-ffd2-48f0-96b4-ae93155284d6/volumes"
Nov 03 21:10:05.215432 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:05.215411    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" path="/var/lib/kubelet/pods/e78b7483-e549-445c-af9e-9b85fdc8f77f/volumes"
Nov 03 21:10:27.368092 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:10:27.368056    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:10:35.853678 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:10:35.853649    4274 scope.go:122] "RemoveContainer" containerID="0fabd614d71c91f427910eba94d739829bca1e1f81e50aa9cabc6b97c4b19e84"
Nov 03 21:10:57.365943 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:10:57.365901    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:11:06.786781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:06.786746    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 21:11:06.787319 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:06.786973    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy" containerID="cri-o://d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b" gracePeriod=30
Nov 03 21:11:12.070723 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.070692    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 21:11:12.238543 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238511    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238715 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238627    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238715 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238662    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238715 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238702    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238839 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238754    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238839 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238785    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:12.238839 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238827    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238881    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238917    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.238986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238942    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\") pod \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\" (UID: \"ffb90bc3-66ab-44d3-81ff-8b95435a177c\") "
Nov 03 21:11:12.239172 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.238990    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:12.239172 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239013    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:12.239172 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239072    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:12.239391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239369    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:12.239489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239467    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.239558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239503    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.239558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239516    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.239558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.239531    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.241459 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.241434    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:11:12.241651 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.241630    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:12.241784 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.241720    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "kube-api-access-4hwss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:12.242400 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.242380    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token" pod "ffb90bc3-66ab-44d3-81ff-8b95435a177c" (UID: "ffb90bc3-66ab-44d3-81ff-8b95435a177c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:12.300942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.300910    4274 generic.go:356] "Generic (PLEG): container finished" podID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerID="d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b" exitCode=0
Nov 03 21:11:12.301117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.300967    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"
Nov 03 21:11:12.301117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.300981    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerDied","Data":"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"}
Nov 03 21:11:12.301117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.301070    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9" event={"ID":"ffb90bc3-66ab-44d3-81ff-8b95435a177c","Type":"ContainerDied","Data":"684a38dbee8e23a5bdca916c85c69c1ada615b7e4185eb270cb06643a73c898a"}
Nov 03 21:11:12.301286 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.301105    4274 scope.go:122] "RemoveContainer" containerID="d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"
Nov 03 21:11:12.311006 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.310987    4274 scope.go:122] "RemoveContainer" containerID="4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"
Nov 03 21:11:12.319216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.319200    4274 scope.go:122] "RemoveContainer" containerID="d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"
Nov 03 21:11:12.319486 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:12.319466    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b\": container with ID starting with d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b not found: ID does not exist" containerID="d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"
Nov 03 21:11:12.319545 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.319494    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b"} err="failed to get container status \"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b\": rpc error: code = NotFound desc = could not find container \"d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b\": container with ID starting with d49f3c32bf777a9363e7ac0cdb570349ee63a7af4264b1fe9af26d95f85bb03b not found: ID does not exist"
Nov 03 21:11:12.319545 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.319509    4274 scope.go:122] "RemoveContainer" containerID="4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"
Nov 03 21:11:12.319747 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:12.319728    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4\": container with ID starting with 4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4 not found: ID does not exist" containerID="4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"
Nov 03 21:11:12.319800 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.319757    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4"} err="failed to get container status \"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4\": rpc error: code = NotFound desc = could not find container \"4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4\": container with ID starting with 4135acae13c59bbc4deecfaa3e85ec7479387a2e4840f8a915e633f5fb3dc6c4 not found: ID does not exist"
Nov 03 21:11:12.340482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.340464    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.340482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.340483    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwss\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-kube-api-access-4hwss\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.340598 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.340497    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.340598 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.340511    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.340598 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.340520    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb90bc3-66ab-44d3-81ff-8b95435a177c-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:12.455274 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.455249    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 21:11:12.461445 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.461425    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-65lv9"]
Nov 03 21:11:12.640338 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.640253    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 21:11:12.669108 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:12.669073    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 21:11:13.213386 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:13.213354    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" path="/var/lib/kubelet/pods/ffb90bc3-66ab-44d3-81ff-8b95435a177c/volumes"
Nov 03 21:11:19.702045 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:19.702001    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 21:11:19.820274 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:19.820244    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 21:11:20.285965 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.285932    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:11:20.286489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286463    4274 cpu_manager.go:414] "containerMap: removing container" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main"
Nov 03 21:11:20.286489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286487    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286505    4274 cpu_manager.go:414] "containerMap: removing container" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286512    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286529    4274 cpu_manager.go:414] "containerMap: removing container" podUID="b0215b97-42ea-404d-9afa-fc82cbbf0740" containerName="maas-api"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286535    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0215b97-42ea-404d-9afa-fc82cbbf0740" containerName="maas-api"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286555    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286563    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286575    4274 cpu_manager.go:414] "containerMap: removing container" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286581    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286598    4274 cpu_manager.go:414] "containerMap: removing container" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286605    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286614    4274 cpu_manager.go:414] "containerMap: removing container" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286623    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286639    4274 cpu_manager.go:414] "containerMap: removing container" podUID="dc238b02-aaa7-418b-990b-ed194b40c725" containerName="maas-api"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286647    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc238b02-aaa7-418b-990b-ed194b40c725" containerName="maas-api"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286672    4274 cpu_manager.go:414] "containerMap: removing container" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main"
Nov 03 21:11:20.286672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286677    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286774    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc238b02-aaa7-418b-990b-ed194b40c725" containerName="maas-api"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286787    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="9734fd48-ac42-436e-8e0a-9e8b3895ddc5" containerName="main"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286794    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="630ef936-48ad-44bc-aa99-e35d2f503099" containerName="main"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286800    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286807    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0215b97-42ea-404d-9afa-fc82cbbf0740" containerName="maas-api"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286820    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac63bf4-034e-4643-9375-c2e01f29f143" containerName="istio-proxy"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286827    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="feff67ee-3424-465b-935f-284debb0350a" containerName="main"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286832    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa51e164-ffd2-48f0-96b4-ae93155284d6" containerName="istio-proxy"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286838    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:20.287322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.286842    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78b7483-e549-445c-af9e-9b85fdc8f77f" containerName="istio-proxy"
Nov 03 21:11:20.291567 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.291548    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.341354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.341326    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndqr\" (UniqueName: \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.341469 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.341368    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.341469 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.341456    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.341604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.341583    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.343700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.343679    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cfcd54-serving-cert\""
Nov 03 21:11:20.346343 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.346308    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:11:20.442248 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.442210    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.442416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.442283    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndqr\" (UniqueName: \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.442416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.442319    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.442416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.442368    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.442416 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:20.442386    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-serving-cert: secret "maas-api-e2e-api-b-cfcd54-serving-cert" not found
Nov 03 21:11:20.442644 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:20.442458    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls podName:b3559366-66db-42c9-87a2-27a543ad6816 nodeName:}" failed. No retries permitted until 2026-11-03 21:11:20.942437261 +0000 UTC m=+2505.869007479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls") pod "maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" (UID: "b3559366-66db-42c9-87a2-27a543ad6816") : secret "maas-api-e2e-api-b-cfcd54-serving-cert" not found
Nov 03 21:11:20.443190 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.443165    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.444993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.444969    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.501946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.501915    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndqr\" (UniqueName: \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.950826 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.950790    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:20.953349 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:20.953324    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") pod \"maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:21.221406 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:21.221318    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:21.643550 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:21.643520    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:11:21.645366 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:11:21.645341    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3559366_66db_42c9_87a2_27a543ad6816.slice/crio-1edcfb1acb1367f60baf719f1f4b1b8e77232b0ac610887f328f15bb0e0f6d69 WatchSource:0}: Error finding container 1edcfb1acb1367f60baf719f1f4b1b8e77232b0ac610887f328f15bb0e0f6d69: Status 404 returned error can't find the container with id 1edcfb1acb1367f60baf719f1f4b1b8e77232b0ac610887f328f15bb0e0f6d69
Nov 03 21:11:21.646934 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:21.646912    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:11:22.364276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:22.364241    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" event={"ID":"b3559366-66db-42c9-87a2-27a543ad6816","Type":"ContainerStarted","Data":"1edcfb1acb1367f60baf719f1f4b1b8e77232b0ac610887f328f15bb0e0f6d69"}
Nov 03 21:11:23.371688 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:23.371656    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" event={"ID":"b3559366-66db-42c9-87a2-27a543ad6816","Type":"ContainerStarted","Data":"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"}
Nov 03 21:11:23.372143 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:23.371790    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:23.496826 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:23.496770    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" podStartSLOduration=2.610415673 podStartE2EDuration="3.496754331s" totalImagesPullingTime="886.338658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:23.494677755 +0000 UTC m=+2508.421247992" watchObservedRunningTime="2026-11-03 21:11:23.496754331 +0000 UTC m=+2508.423324567"
Nov 03 21:11:24.702928 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:24.702882    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing" containerID="cri-o://032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154" gracePeriod=25
Nov 03 21:11:24.820917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:24.820855    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing" containerID="cri-o://d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b" gracePeriod=25
Nov 03 21:11:27.365990 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:27.365956    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:11:27.660894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:27.660803    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main" containerID="cri-o://fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5" gracePeriod=45
Nov 03 21:11:27.688637 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:27.688604    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" containerID="cri-o://300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951" gracePeriod=45
Nov 03 21:11:27.857402 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:27.857364    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Nov 03 21:11:27.981698 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:27.981666    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 21:11:27.984542 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:27.984521    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 21:11:28.034626 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034598    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.034757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034638    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.034757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034693    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.034757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034726    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.034878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034825    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\" (UniqueName: \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.034878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034854    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\" (UniqueName: \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.034962 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034907    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.034962 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034937    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035105 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034962    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\" (UniqueName: \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.035105 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034991    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.035105 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035024    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.035273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035105    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\") pod \"0036e8ea-399b-42ba-96dd-f2034df77690\" (UID: \"0036e8ea-399b-42ba-96dd-f2034df77690\") "
Nov 03 21:11:28.035273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035215    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\") pod \"b8969180-8963-449a-b5cd-93a6524a864c\" (UID: \"b8969180-8963-449a-b5cd-93a6524a864c\") "
Nov 03 21:11:28.035447 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.034959    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035447 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035172    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035447 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035359    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035606 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035564    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035747 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035724    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.035970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035923    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.035970 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035951    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.036148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.035989    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.036148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.036003    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.036148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.036017    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.036148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.036050    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.037277 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037254    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "kube-api-access-9txwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:28.037399 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037363    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:28.037399 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037385    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm" pod "0036e8ea-399b-42ba-96dd-f2034df77690" (UID: "0036e8ea-399b-42ba-96dd-f2034df77690"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.037539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037521    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.037585 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037538    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "kube-api-access-jkqrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:28.037635 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.037618    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs" pod "b8969180-8963-449a-b5cd-93a6524a864c" (UID: "b8969180-8963-449a-b5cd-93a6524a864c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:28.137378 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137340    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8969180-8963-449a-b5cd-93a6524a864c-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.137378 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137382    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0036e8ea-399b-42ba-96dd-f2034df77690-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.137378 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137399    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txwg\" (UniqueName: \"kubernetes.io/projected/0036e8ea-399b-42ba-96dd-f2034df77690-kube-api-access-9txwg\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.137672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137415    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqrn\" (UniqueName: \"kubernetes.io/projected/b8969180-8963-449a-b5cd-93a6524a864c-kube-api-access-jkqrn\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.137672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137429    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0036e8ea-399b-42ba-96dd-f2034df77690-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.137672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.137444    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8969180-8963-449a-b5cd-93a6524a864c-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.405777 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.405746    4274 generic.go:356] "Generic (PLEG): container finished" podID="0036e8ea-399b-42ba-96dd-f2034df77690" containerID="fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5" exitCode=0
Nov 03 21:11:28.406266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.405787    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"
Nov 03 21:11:28.406266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.405825    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" event={"ID":"0036e8ea-399b-42ba-96dd-f2034df77690","Type":"ContainerDied","Data":"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"}
Nov 03 21:11:28.406266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.405864    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz" event={"ID":"0036e8ea-399b-42ba-96dd-f2034df77690","Type":"ContainerDied","Data":"b0f76171757034b07754fb4917179fc55ad7c808b9ecf9c1c679addab7dc9338"}
Nov 03 21:11:28.406266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.405885    4274 scope.go:122] "RemoveContainer" containerID="fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"
Nov 03 21:11:28.407451 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.407428    4274 generic.go:356] "Generic (PLEG): container finished" podID="b8969180-8963-449a-b5cd-93a6524a864c" containerID="300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951" exitCode=0
Nov 03 21:11:28.407578 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.407460    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" event={"ID":"b8969180-8963-449a-b5cd-93a6524a864c","Type":"ContainerDied","Data":"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"}
Nov 03 21:11:28.407578 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.407479    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj" event={"ID":"b8969180-8963-449a-b5cd-93a6524a864c","Type":"ContainerDied","Data":"b143d54665b3c9fcf82e18624cbb9cc1b6a57d6e2ada9b87877b73785d6e6700"}
Nov 03 21:11:28.407578 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.407488    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"
Nov 03 21:11:28.417500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.417353    4274 scope.go:122] "RemoveContainer" containerID="fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"
Nov 03 21:11:28.417672 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:28.417650    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5\": container with ID starting with fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5 not found: ID does not exist" containerID="fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"
Nov 03 21:11:28.417746 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.417682    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5"} err="failed to get container status \"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5\": rpc error: code = NotFound desc = could not find container \"fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5\": container with ID starting with fee3f31e81976e50a93da249a5a6b642ddfee5457d072510b950b40bc4cc0da5 not found: ID does not exist"
Nov 03 21:11:28.417746 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.417705    4274 scope.go:122] "RemoveContainer" containerID="300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"
Nov 03 21:11:28.425837 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.425819    4274 scope.go:122] "RemoveContainer" containerID="300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"
Nov 03 21:11:28.426113 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:28.426091    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951\": container with ID starting with 300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951 not found: ID does not exist" containerID="300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"
Nov 03 21:11:28.426185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.426120    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951"} err="failed to get container status \"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951\": rpc error: code = NotFound desc = could not find container \"300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951\": container with ID starting with 300f109f848c8cc4fb9fbbc7c33a4b7b21446ba1e0344ad854a9c5079ec73951 not found: ID does not exist"
Nov 03 21:11:28.558772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.558738    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 21:11:28.566392 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.566370    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/premium-w0-1e6fe1-kserve-6b96764b74-p2rxj"]
Nov 03 21:11:28.670084 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.670005    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 21:11:28.674528 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:28.674505    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-2-w0-1e6fe1-kserve-747677f6c8-68qvz"]
Nov 03 21:11:29.213091 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:29.213050    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" path="/var/lib/kubelet/pods/0036e8ea-399b-42ba-96dd-f2034df77690/volumes"
Nov 03 21:11:29.213576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:29.213555    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8969180-8963-449a-b5cd-93a6524a864c" path="/var/lib/kubelet/pods/b8969180-8963-449a-b5cd-93a6524a864c/volumes"
Nov 03 21:11:29.382882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:29.382846    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:11:32.096664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:32.096627    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:32.097082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:32.096631    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:42.115059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:42.113118    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:42.115059 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:42.113578    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:42.278056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:42.277140    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:11:42.278056 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:42.277220    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:11:44.968120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:44.968030    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 21:11:45.074248 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.074224    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 21:11:45.115971 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.115945    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\") pod \"ce54e08b-44d3-408b-810f-c4d555985350\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") "
Nov 03 21:11:45.116122 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.116110    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\" (UniqueName: \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\") pod \"ce54e08b-44d3-408b-810f-c4d555985350\" (UID: \"ce54e08b-44d3-408b-810f-c4d555985350\") "
Nov 03 21:11:45.116345 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.116319    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume" pod "ce54e08b-44d3-408b-810f-c4d555985350" (UID: "ce54e08b-44d3-408b-810f-c4d555985350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:45.116468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.116453    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce54e08b-44d3-408b-810f-c4d555985350-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:45.118092 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.118069    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc" pod "ce54e08b-44d3-408b-810f-c4d555985350" (UID: "ce54e08b-44d3-408b-810f-c4d555985350"). InnerVolumeSpecName "kube-api-access-fb7qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:45.217140 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.217107    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\") pod \"6273b213-7f67-433c-97d7-8178be69f048\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") "
Nov 03 21:11:45.217313 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.217283    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\" (UniqueName: \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\") pod \"6273b213-7f67-433c-97d7-8178be69f048\" (UID: \"6273b213-7f67-433c-97d7-8178be69f048\") "
Nov 03 21:11:45.217549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.217520    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume" pod "6273b213-7f67-433c-97d7-8178be69f048" (UID: "6273b213-7f67-433c-97d7-8178be69f048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:11:45.217627 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.217587    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb7qc\" (UniqueName: \"kubernetes.io/projected/ce54e08b-44d3-408b-810f-c4d555985350-kube-api-access-fb7qc\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:45.219457 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.219393    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj" pod "6273b213-7f67-433c-97d7-8178be69f048" (UID: "6273b213-7f67-433c-97d7-8178be69f048"). InnerVolumeSpecName "kube-api-access-hb7kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:45.319104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.319073    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb7kj\" (UniqueName: \"kubernetes.io/projected/6273b213-7f67-433c-97d7-8178be69f048-kube-api-access-hb7kj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:45.319104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.319100    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6273b213-7f67-433c-97d7-8178be69f048-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:45.550073 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.550011    4274 generic.go:356] "Generic (PLEG): container finished" podID="6273b213-7f67-433c-97d7-8178be69f048" containerID="d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b" exitCode=0
Nov 03 21:11:45.550264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.550066    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" event={"ID":"6273b213-7f67-433c-97d7-8178be69f048","Type":"ContainerDied","Data":"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"}
Nov 03 21:11:45.550264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.550084    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"
Nov 03 21:11:45.550264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.550158    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k" event={"ID":"6273b213-7f67-433c-97d7-8178be69f048","Type":"ContainerDied","Data":"5ce783e7a4b5e207edac2801f0303f9dfcce97dd050b85969ca468860c1f83e8"}
Nov 03 21:11:45.550264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.550189    4274 scope.go:122] "RemoveContainer" containerID="d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"
Nov 03 21:11:45.551512 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.551488    4274 generic.go:356] "Generic (PLEG): container finished" podID="ce54e08b-44d3-408b-810f-c4d555985350" containerID="032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154" exitCode=0
Nov 03 21:11:45.551609 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.551557    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"
Nov 03 21:11:45.551670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.551545    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" event={"ID":"ce54e08b-44d3-408b-810f-c4d555985350","Type":"ContainerDied","Data":"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"}
Nov 03 21:11:45.551725 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.551675    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc" event={"ID":"ce54e08b-44d3-408b-810f-c4d555985350","Type":"ContainerDied","Data":"b679b4f4d2dad8ee83bdf7a5d47f9cf0f3c0f442889b136597c5186905f4737a"}
Nov 03 21:11:45.560636 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.560621    4274 scope.go:122] "RemoveContainer" containerID="d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"
Nov 03 21:11:45.560875 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:45.560857    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b\": container with ID starting with d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b not found: ID does not exist" containerID="d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"
Nov 03 21:11:45.560941 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.560880    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b"} err="failed to get container status \"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b\": rpc error: code = NotFound desc = could not find container \"d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b\": container with ID starting with d149b141a4c22f03427bf2d3cd1f265f82a726d50d3816421c941ff5df96c43b not found: ID does not exist"
Nov 03 21:11:45.560941 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.560897    4274 scope.go:122] "RemoveContainer" containerID="032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"
Nov 03 21:11:45.569224 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.569206    4274 scope.go:122] "RemoveContainer" containerID="032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"
Nov 03 21:11:45.569453 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:45.569435    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154\": container with ID starting with 032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154 not found: ID does not exist" containerID="032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"
Nov 03 21:11:45.569497 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.569458    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154"} err="failed to get container status \"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154\": rpc error: code = NotFound desc = could not find container \"032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154\": container with ID starting with 032f3c48030c6214f0bff84d389067e2640083a36e04561b031244acca774154 not found: ID does not exist"
Nov 03 21:11:45.703116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.703086    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 21:11:45.707572 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.707547    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1e6fe1-7f5cb5f5c5-6z7mc"]
Nov 03 21:11:45.810490 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.810413    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 21:11:45.817123 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:45.817098    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1e6fe1-8c49bd5ff-q6j7k"]
Nov 03 21:11:47.214250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:47.214222    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6273b213-7f67-433c-97d7-8178be69f048" path="/var/lib/kubelet/pods/6273b213-7f67-433c-97d7-8178be69f048/volumes"
Nov 03 21:11:47.214635 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:47.214621    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce54e08b-44d3-408b-810f-c4d555985350" path="/var/lib/kubelet/pods/ce54e08b-44d3-408b-810f-c4d555985350/volumes"
Nov 03 21:11:55.882945 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.882903    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:11:55.884378 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884348    4274 cpu_manager.go:414] "containerMap: removing container" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main"
Nov 03 21:11:55.884545 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884531    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main"
Nov 03 21:11:55.884667 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884653    4274 cpu_manager.go:414] "containerMap: removing container" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:55.884771 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884751    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb90bc3-66ab-44d3-81ff-8b95435a177c" containerName="istio-proxy"
Nov 03 21:11:55.884888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884874    4274 cpu_manager.go:414] "containerMap: removing container" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing"
Nov 03 21:11:55.884964 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.884954    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing"
Nov 03 21:11:55.885079 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885053    4274 cpu_manager.go:414] "containerMap: removing container" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing"
Nov 03 21:11:55.885188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885174    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing"
Nov 03 21:11:55.885267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885258    4274 cpu_manager.go:414] "containerMap: removing container" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main"
Nov 03 21:11:55.885338 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885327    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main"
Nov 03 21:11:55.885576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885559    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce54e08b-44d3-408b-810f-c4d555985350" containerName="payload-processing"
Nov 03 21:11:55.885691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885679    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6273b213-7f67-433c-97d7-8178be69f048" containerName="payload-pre-processing"
Nov 03 21:11:55.885787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885776    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="0036e8ea-399b-42ba-96dd-f2034df77690" containerName="main"
Nov 03 21:11:55.885885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.885875    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8969180-8963-449a-b5cd-93a6524a864c" containerName="main"
Nov 03 21:11:55.893301 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.893275    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.948500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.948464    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.948878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.948850    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbmm\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.949163 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.949085    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950222    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950448 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950419    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950552 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950472    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950608 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950552    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950608 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950597    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.950691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.950626    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:55.974448 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:55.974414    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:11:56.017554 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.017522    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c971dc-openshift-default-dockercfg-flh6l\""
Nov 03 21:11:56.051499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051464    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051558    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051584    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051603    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051652    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051684    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051956 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051708    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051956 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051791    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.051956 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.051817    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbmm\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.052178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.052018    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.052178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.052119    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.052298 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.052262    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.052402 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.052375    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.052633 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.052613    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.054014 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.053990    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.054158 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.054142    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.112662 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.112628    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbmm\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.113489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.113456    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\") pod \"e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") " pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.215785 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.215707    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:56.633376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.633351    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:11:56.635240 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:11:56.635208    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcae569_67ea_4a3b_95e6_085806de6f58.slice/crio-d36a1c0136c06a5bc94375656e483c5ce107b0fb5256899d074f7e2406f0462d WatchSource:0}: Error finding container d36a1c0136c06a5bc94375656e483c5ce107b0fb5256899d074f7e2406f0462d: Status 404 returned error can't find the container with id d36a1c0136c06a5bc94375656e483c5ce107b0fb5256899d074f7e2406f0462d
Nov 03 21:11:56.637818 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.637788    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:56.637891 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.637855    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:56.637891 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:56.637883    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:57.365074 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:11:57.365014    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:11:57.634101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:57.633967    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerStarted","Data":"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"}
Nov 03 21:11:57.634279 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:57.634112    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerStarted","Data":"d36a1c0136c06a5bc94375656e483c5ce107b0fb5256899d074f7e2406f0462d"}
Nov 03 21:11:57.747896 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:57.747847    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podStartSLOduration=2.747834161 podStartE2EDuration="2.747834161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:57.747333297 +0000 UTC m=+2542.673903535" watchObservedRunningTime="2026-11-03 21:11:57.747834161 +0000 UTC m=+2542.674404398"
Nov 03 21:11:58.215885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:58.215851    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:58.221294 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:58.221270    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:58.639827 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:58.639795    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:58.640816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:58.640795    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:11:59.187465 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.187427    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:11:59.191318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.191294    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.243212 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.243189    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f1855-openshift-default-dockercfg-n94fj\""
Nov 03 21:11:59.290675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290637    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.290878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290697    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.290878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290733    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.290878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290799    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.290878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290824    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.291150 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.290911    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5ps\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.291209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.291139    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.291209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.291198    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.291310 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.291240    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.292458 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.292432    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:11:59.392318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392282    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392322    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392382    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392405    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392487    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5ps\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392566 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392531    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392783 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392577    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392783 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392613    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392783 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392750    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392794    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.392946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.392889    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.393108 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.393082    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.393185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.393152    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.393410 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.393391    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.394773 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.394752    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.395106 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.395085    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.449911 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.449848    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.449911 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.449900    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5ps\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.505264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.505235    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:11:59.927666 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.927639    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:11:59.929194 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:11:59.929154    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74b724f_1f77_41d3_beef_430899abeb4d.slice/crio-92a8842818ab3dc8e3a9d9af0da4b957c4fd4e24797ff4ebf131281bc9fc3b7c WatchSource:0}: Error finding container 92a8842818ab3dc8e3a9d9af0da4b957c4fd4e24797ff4ebf131281bc9fc3b7c: Status 404 returned error can't find the container with id 92a8842818ab3dc8e3a9d9af0da4b957c4fd4e24797ff4ebf131281bc9fc3b7c
Nov 03 21:11:59.931803 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.931772    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:59.931874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.931840    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:59.931874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:11:59.931867    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:12:00.674414 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:00.674370    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" event={"ID":"c74b724f-1f77-41d3-beef-430899abeb4d","Type":"ContainerStarted","Data":"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"}
Nov 03 21:12:00.674632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:00.674439    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" event={"ID":"c74b724f-1f77-41d3-beef-430899abeb4d","Type":"ContainerStarted","Data":"92a8842818ab3dc8e3a9d9af0da4b957c4fd4e24797ff4ebf131281bc9fc3b7c"}
Nov 03 21:12:00.865619 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:00.865554    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podStartSLOduration=1.865531601 podStartE2EDuration="1.865531601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:00.865335613 +0000 UTC m=+2545.791905851" watchObservedRunningTime="2026-11-03 21:12:00.865531601 +0000 UTC m=+2545.792101839"
Nov 03 21:12:01.505558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:01.505524    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:12:01.506947 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:01.506916    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:12:01.507084 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:01.506974    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:12:02.506338 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:02.506303    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:12:02.506749 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:02.506370    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:12:03.506487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:03.506454    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:12:03.506928 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:03.506505    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:12:04.505728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:04.505689    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:12:04.505917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:04.505750    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:12:05.506074 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:05.506019    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:12:05.506456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:05.506094    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:12:06.510185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:06.510154    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:12:06.510630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:06.510616    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:12:06.511247 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:06.511224    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:12:08.665875 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.665835    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:12:08.669350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.669328    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.718007 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.717975    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c971dc\""
Nov 03 21:12:08.746415 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.746375    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g444w\" (UniqueName: \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.746733 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.746702    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.764836 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.764803    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c971dc-dockercfg-746cc\""
Nov 03 21:12:08.768931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.768898    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:12:08.816737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.816705    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:12:08.823152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.823123    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.847736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.847701    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g444w\" (UniqueName: \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.847926 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.847830    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.847926 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.847901    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.848057 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.847954    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6w9\" (UniqueName: \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.849659 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.849631    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.866440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.866408    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:12:08.910513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.910481    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g444w\" (UniqueName: \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\") pod \"payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:08.949434 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.949357    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6w9\" (UniqueName: \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.949565 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.949509    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.950200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.950182    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:08.983295 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:08.983263    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:09.046385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.045871    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6w9\" (UniqueName: \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\") pod \"payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:09.145303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.145277    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:09.382612 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.382569    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:12:09.383599 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:12:09.383573    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a39c18_5bcf_4ddf_b436_9b507c712d74.slice/crio-629ba6d93346e6534c37d6d4fc8ebf4c1be91806ad482682bd0b741dde8ab28c WatchSource:0}: Error finding container 629ba6d93346e6534c37d6d4fc8ebf4c1be91806ad482682bd0b741dde8ab28c: Status 404 returned error can't find the container with id 629ba6d93346e6534c37d6d4fc8ebf4c1be91806ad482682bd0b741dde8ab28c
Nov 03 21:12:09.516001 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.515979    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:12:09.519073 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:12:09.519021    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c871ba_1376_444f_9242_0e719398a2c0.slice/crio-3a481028c59a8902eec37840d3d7a377fb984f96e1b5ed830079e50afdaac296 WatchSource:0}: Error finding container 3a481028c59a8902eec37840d3d7a377fb984f96e1b5ed830079e50afdaac296: Status 404 returned error can't find the container with id 3a481028c59a8902eec37840d3d7a377fb984f96e1b5ed830079e50afdaac296
Nov 03 21:12:09.723016 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.722979    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" event={"ID":"a3c871ba-1376-444f-9242-0e719398a2c0","Type":"ContainerStarted","Data":"88fe25b38fcbc1298fbfe1add06a5d95bf89f3135da5ed384709cebd6da0e59a"}
Nov 03 21:12:09.723461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.723065    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" event={"ID":"a3c871ba-1376-444f-9242-0e719398a2c0","Type":"ContainerStarted","Data":"3a481028c59a8902eec37840d3d7a377fb984f96e1b5ed830079e50afdaac296"}
Nov 03 21:12:09.723461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.723168    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:09.724604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.724580    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" event={"ID":"95a39c18-5bcf-4ddf-b436-9b507c712d74","Type":"ContainerStarted","Data":"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"}
Nov 03 21:12:09.724707 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.724618    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" event={"ID":"95a39c18-5bcf-4ddf-b436-9b507c712d74","Type":"ContainerStarted","Data":"629ba6d93346e6534c37d6d4fc8ebf4c1be91806ad482682bd0b741dde8ab28c"}
Nov 03 21:12:09.724707 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.724654    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:09.833427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.833340    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" podStartSLOduration=1.833327861 podStartE2EDuration="1.833327861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:09.831615191 +0000 UTC m=+2554.758185427" watchObservedRunningTime="2026-11-03 21:12:09.833327861 +0000 UTC m=+2554.759898098"
Nov 03 21:12:09.947126 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:09.947061    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" podStartSLOduration=1.9470260480000001 podStartE2EDuration="1.947026048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:09.946996962 +0000 UTC m=+2554.873567200" watchObservedRunningTime="2026-11-03 21:12:09.947026048 +0000 UTC m=+2554.873596285"
Nov 03 21:12:20.733296 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:20.733268    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:12:20.735549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:20.733337    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:12:27.366003 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:12:27.365963    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:12:44.560102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.560069    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:12:44.565316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.565293    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.587994    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588143 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588027    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588143 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588129    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588216 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588190    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588250    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588279    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588336 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588303    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588336 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588328    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.588443 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.588429    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wkth\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.663732 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.663701    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:12:44.689371 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689340    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wkth\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689507 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689455    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689507 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689484    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689594 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689529    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689642 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689598    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689695 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689638    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689695 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689669    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689702    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689727    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.689978 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.689948    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.690302 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.690274    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.690382 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.690319    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.690529 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.690501    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.690781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.690758    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.692290 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.692265    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.692708 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.692683    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.746709 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.746684    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.746805 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.746771    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wkth\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:44.878674 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:44.878597    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:45.298516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.298458    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:12:45.299116 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:12:45.299083    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde804723_9f46_4882_b35d_e9e36a73880f.slice/crio-e30b8bde3ed53c3d43d1f1c9a6b686eeef0b30790f0155a07877720bcc0e7ac2 WatchSource:0}: Error finding container e30b8bde3ed53c3d43d1f1c9a6b686eeef0b30790f0155a07877720bcc0e7ac2: Status 404 returned error can't find the container with id e30b8bde3ed53c3d43d1f1c9a6b686eeef0b30790f0155a07877720bcc0e7ac2
Nov 03 21:12:45.302995 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.302959    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:12:45.303095 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.303053    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:12:45.303143 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.303094    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:12:45.930844 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.930793    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" event={"ID":"de804723-9f46-4882-b35d-e9e36a73880f","Type":"ContainerStarted","Data":"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"}
Nov 03 21:12:45.930844 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:45.930849    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" event={"ID":"de804723-9f46-4882-b35d-e9e36a73880f","Type":"ContainerStarted","Data":"e30b8bde3ed53c3d43d1f1c9a6b686eeef0b30790f0155a07877720bcc0e7ac2"}
Nov 03 21:12:46.098016 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:46.097935    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podStartSLOduration=2.097916525 podStartE2EDuration="2.097916525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:46.097506243 +0000 UTC m=+2591.024076482" watchObservedRunningTime="2026-11-03 21:12:46.097916525 +0000 UTC m=+2591.024486762"
Nov 03 21:12:46.879246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:46.879203    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:46.880891 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:46.880861    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 21:12:46.881048 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:46.880911    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 21:12:47.894114 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:47.888864    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 21:12:47.894114 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:47.888928    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 21:12:48.879439 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:48.879403    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 21:12:48.879719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:48.879691    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 21:12:49.879492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:49.879456    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 21:12:49.879909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:49.879521    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 21:12:50.879923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:50.879884    4274 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 21:12:50.880385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:50.879952    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 21:12:51.884100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:51.884064    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:51.884623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:51.884373    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:51.884999 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:51.884984    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:12:54.949168 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:54.949138    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 21:12:54.949603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:54.949356    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" containerID="cri-o://8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816" gracePeriod=30
Nov 03 21:12:56.483178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:56.483144    4274 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:12:56.483628 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:12:56.483213    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:12:57.371438 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:12:57.371401    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:13:00.218870 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.218811    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 21:13:00.347184 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347146    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347222    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347353    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347386    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347420    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347455    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347495    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347532    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347573    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\") pod \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\" (UID: \"158e614f-f439-4a57-8c6e-a8d9ba3e1778\") "
Nov 03 21:13:00.347704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347568    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:00.347704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347646    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:00.347926 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.347904    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:00.348191 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.348166    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.348390 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.348376    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.348500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.348482    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.348500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.348163    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:00.348686 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.348237    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:00.350010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.349984    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:00.350239 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.350217    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:13:00.350350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.350256    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:00.350350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.350285    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5" pod "158e614f-f439-4a57-8c6e-a8d9ba3e1778" (UID: "158e614f-f439-4a57-8c6e-a8d9ba3e1778"). InnerVolumeSpecName "kube-api-access-mxbm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:00.449677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449641    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxbm5\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-kube-api-access-mxbm5\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.449677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449669    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.449677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449680    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.449965 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449689    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.449965 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449698    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:00.449965 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:00.449706    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158e614f-f439-4a57-8c6e-a8d9ba3e1778-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:01.045499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.045456    4274 generic.go:356] "Generic (PLEG): container finished" podID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerID="8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816" exitCode=0
Nov 03 21:13:01.045672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.045523    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"
Nov 03 21:13:01.045672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.045533    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerDied","Data":"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"}
Nov 03 21:13:01.045672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.045614    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf" event={"ID":"158e614f-f439-4a57-8c6e-a8d9ba3e1778","Type":"ContainerDied","Data":"71c47bffeab99f5699625c76c96b800674a9dfcaa3198e7103b7f8dc3ea86da1"}
Nov 03 21:13:01.045672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.045641    4274 scope.go:122] "RemoveContainer" containerID="8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"
Nov 03 21:13:01.055619 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.055599    4274 scope.go:122] "RemoveContainer" containerID="943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"
Nov 03 21:13:01.063585 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.063568    4274 scope.go:122] "RemoveContainer" containerID="8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"
Nov 03 21:13:01.063816 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:13:01.063798    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816\": container with ID starting with 8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816 not found: ID does not exist" containerID="8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"
Nov 03 21:13:01.063864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.063822    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816"} err="failed to get container status \"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816\": rpc error: code = NotFound desc = could not find container \"8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816\": container with ID starting with 8b66bcd2128adb4cb321034a538c19c80cf0759fb4a62ac7f527f539396a1816 not found: ID does not exist"
Nov 03 21:13:01.063864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.063839    4274 scope.go:122] "RemoveContainer" containerID="943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"
Nov 03 21:13:01.064100 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:13:01.064081    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3\": container with ID starting with 943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3 not found: ID does not exist" containerID="943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"
Nov 03 21:13:01.064151 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.064107    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3"} err="failed to get container status \"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3\": rpc error: code = NotFound desc = could not find container \"943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3\": container with ID starting with 943f89d3c8274da8d53d45675ba48c85a9d9a16daadf9339cd259fc011c87dc3 not found: ID does not exist"
Nov 03 21:13:01.249342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.249311    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 21:13:01.295270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:01.295244    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-4gthf"]
Nov 03 21:13:03.213089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:03.213058    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" path="/var/lib/kubelet/pods/158e614f-f439-4a57-8c6e-a8d9ba3e1778/volumes"
Nov 03 21:13:27.366152 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:13:27.366110    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:13:28.782825 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:28.782309    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": read tcp 10.133.0.2:50314->10.133.0.107:15021: read: connection reset by peer" start-of-body=
Nov 03 21:13:28.782825 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:28.782372    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": read tcp 10.133.0.2:50314->10.133.0.107:15021: read: connection reset by peer"
Nov 03 21:13:29.234148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.234065    4274 generic.go:356] "Generic (PLEG): container finished" podID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerID="27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e" exitCode=0
Nov 03 21:13:29.234321 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.234140    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerDied","Data":"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"}
Nov 03 21:13:29.234561 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.234544    4274 scope.go:122] "RemoveContainer" containerID="27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"
Nov 03 21:13:29.236021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.235997    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:29.236120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.236080    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:29.236120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:29.236116    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:30.241451 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:30.241416    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerStarted","Data":"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"}
Nov 03 21:13:31.216204 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:31.216167    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:13:31.216468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:31.216322    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:13:31.216468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:31.216365    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 21:13:32.216217 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:32.216176    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:13:32.216618 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:32.216245    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 21:13:33.216489 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:33.216447    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:13:33.217005 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:33.216509    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 21:13:34.216429 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:34.216397    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:13:34.216611 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:34.216468    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 21:13:35.216520 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:35.216486    4274 patch_prober.go:29] interesting pod/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:13:35.216718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:35.216552    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 21:13:36.220474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:36.220403    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:13:36.220915 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:36.220881    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:13:36.221663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:13:36.221644    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:13:57.365606 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:13:57.365569    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:14:27.365882 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:14:27.365840    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:14:57.364796 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:14:57.364759    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:15:00.207328 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.207294    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:00.207866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.207847    4274 cpu_manager.go:414] "containerMap: removing container" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.207944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.207869    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.207944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.207882    4274 cpu_manager.go:414] "containerMap: removing container" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.207944 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.207887    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.208132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.208008    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.211496 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.211473    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:00.211707 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.211686    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.212132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.212113    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="158e614f-f439-4a57-8c6e-a8d9ba3e1778" containerName="istio-proxy"
Nov 03 21:15:00.215186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.215172    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.300261 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.300226    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqsg\" (UniqueName: \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.300523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.300499    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dndj\" (UniqueName: \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.300654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.300630    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.300794 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.300663    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.327281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.327247    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:00.365259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.365231    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:15:00.412993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.403018    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:00.413192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.410514    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.413192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.411679    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.413192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.413161    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.413422 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.413396    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqsg\" (UniqueName: \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.413807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.413787    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dndj\" (UniqueName: \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.414503 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.414482    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.471079 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.471017    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqsg\" (UniqueName: \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\") pod \"maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.471213 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.471101    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dndj\" (UniqueName: \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\") pod \"maas-api-key-cleanup-29895675-swcj2\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.526117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.526091    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:00.531533 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.531508    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:00.903211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.903185    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:00.904126 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:15:00.904059    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd704c925_67cb_480d_8130_2987c888b484.slice/crio-0f301fcbc798e723b03925d211b7220fd7423ce235bbd5ea040cae5499a09d78 WatchSource:0}: Error finding container 0f301fcbc798e723b03925d211b7220fd7423ce235bbd5ea040cae5499a09d78: Status 404 returned error can't find the container with id 0f301fcbc798e723b03925d211b7220fd7423ce235bbd5ea040cae5499a09d78
Nov 03 21:15:00.942563 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:00.942535    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:00.943251 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:15:00.943220    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f781bc_3dfa_45bf_b7ce_8d19954a8aa0.slice/crio-966a3f3efd4d31a21c23d7fe63c948449f4039abdd08d0b03e0f5469384f455a WatchSource:0}: Error finding container 966a3f3efd4d31a21c23d7fe63c948449f4039abdd08d0b03e0f5469384f455a: Status 404 returned error can't find the container with id 966a3f3efd4d31a21c23d7fe63c948449f4039abdd08d0b03e0f5469384f455a
Nov 03 21:15:01.788853 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.788818    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerStarted","Data":"923a4780a91f493c3527a336007dc70f5c6109d614af5e592ccc84074c5dbc07"}
Nov 03 21:15:01.789337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.788879    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerStarted","Data":"966a3f3efd4d31a21c23d7fe63c948449f4039abdd08d0b03e0f5469384f455a"}
Nov 03 21:15:01.790758 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.790730    4274 generic.go:356] "Generic (PLEG): container finished" podID="d704c925-67cb-480d-8130-2987c888b484" containerID="a61c90cc9d92516e697a647187c8ebb9512cea2ff91f8380ead2271ef4e36c77" exitCode=60
Nov 03 21:15:01.790864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.790788    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerDied","Data":"a61c90cc9d92516e697a647187c8ebb9512cea2ff91f8380ead2271ef4e36c77"}
Nov 03 21:15:01.790864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.790829    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerStarted","Data":"0f301fcbc798e723b03925d211b7220fd7423ce235bbd5ea040cae5499a09d78"}
Nov 03 21:15:01.791078 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.791066    4274 scope.go:122] "RemoveContainer" containerID="a61c90cc9d92516e697a647187c8ebb9512cea2ff91f8380ead2271ef4e36c77"
Nov 03 21:15:01.898705 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:01.898651    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" podStartSLOduration=1.89863732 podStartE2EDuration="1.89863732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.896440495 +0000 UTC m=+2726.823010732" watchObservedRunningTime="2026-11-03 21:15:01.89863732 +0000 UTC m=+2726.825207590"
Nov 03 21:15:02.798490 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.798457    4274 generic.go:356] "Generic (PLEG): container finished" podID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerID="923a4780a91f493c3527a336007dc70f5c6109d614af5e592ccc84074c5dbc07" exitCode=60
Nov 03 21:15:02.798931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.798544    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerDied","Data":"923a4780a91f493c3527a336007dc70f5c6109d614af5e592ccc84074c5dbc07"}
Nov 03 21:15:02.798931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.798802    4274 scope.go:122] "RemoveContainer" containerID="923a4780a91f493c3527a336007dc70f5c6109d614af5e592ccc84074c5dbc07"
Nov 03 21:15:02.800661 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.800636    4274 generic.go:356] "Generic (PLEG): container finished" podID="d704c925-67cb-480d-8130-2987c888b484" containerID="9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91" exitCode=60
Nov 03 21:15:02.800758 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.800698    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerDied","Data":"9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91"}
Nov 03 21:15:02.800758 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.800745    4274 scope.go:122] "RemoveContainer" containerID="a61c90cc9d92516e697a647187c8ebb9512cea2ff91f8380ead2271ef4e36c77"
Nov 03 21:15:02.800954 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:02.800939    4274 scope.go:122] "RemoveContainer" containerID="9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91"
Nov 03 21:15:02.801208 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:02.801183    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-swcj2_odh-ai-gateway-infra(d704c925-67cb-480d-8130-2987c888b484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" podUID="d704c925-67cb-480d-8130-2987c888b484"
Nov 03 21:15:03.809116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:03.809082    4274 generic.go:356] "Generic (PLEG): container finished" podID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerID="e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628" exitCode=60
Nov 03 21:15:03.809616 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:03.809166    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerDied","Data":"e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628"}
Nov 03 21:15:03.809616 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:03.809238    4274 scope.go:122] "RemoveContainer" containerID="923a4780a91f493c3527a336007dc70f5c6109d614af5e592ccc84074c5dbc07"
Nov 03 21:15:03.809616 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:03.809502    4274 scope.go:122] "RemoveContainer" containerID="e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628"
Nov 03 21:15:03.809857 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:03.809830    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk_odh-ai-gateway-infra(83f781bc-3dfa-45bf-b7ce-8d19954a8aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0"
Nov 03 21:15:03.811161 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:03.811144    4274 scope.go:122] "RemoveContainer" containerID="9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91"
Nov 03 21:15:03.811417 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:03.811399    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-swcj2_odh-ai-gateway-infra(d704c925-67cb-480d-8130-2987c888b484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" podUID="d704c925-67cb-480d-8130-2987c888b484"
Nov 03 21:15:04.818934 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:04.818904    4274 scope.go:122] "RemoveContainer" containerID="e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628"
Nov 03 21:15:04.819428 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:04.819234    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk_odh-ai-gateway-infra(83f781bc-3dfa-45bf-b7ce-8d19954a8aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0"
Nov 03 21:15:15.213642 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:15.213573    4274 scope.go:122] "RemoveContainer" containerID="9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91"
Nov 03 21:15:15.786691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:15.786657    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 21:15:15.786850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:15.786832    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery" containerID="cri-o://e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21" gracePeriod=30
Nov 03 21:15:15.879549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:15.879518    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerStarted","Data":"5e34ce68d27aad86bfe1ee3bfb922eafe9c85bd2672923499d9f16732f8fe12d"}
Nov 03 21:15:16.014027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.013970    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" podStartSLOduration=16.013954458 podStartE2EDuration="16.013954458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:16.011350179 +0000 UTC m=+2740.937920418" watchObservedRunningTime="2026-11-03 21:15:16.013954458 +0000 UTC m=+2740.940524712"
Nov 03 21:15:16.082183 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.082161    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 21:15:16.187477 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187450    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187503    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187651 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187623    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187725    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187798 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187748    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187810    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.187858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.187852    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\") pod \"780bf404-4247-4e97-8c62-bec19bb76e2e\" (UID: \"780bf404-4247-4e97-8c62-bec19bb76e2e\") "
Nov 03 21:15:16.188314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.188281    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:16.188468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.188444    4274 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-ca-configmap\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.189802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.189778    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:16.189802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.189795    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:16.190163 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.190144    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "kube-api-access-wj2df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:16.190242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.190183    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:16.190296 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.190248    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:16.190660 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.190645    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts" pod "780bf404-4247-4e97-8c62-bec19bb76e2e" (UID: "780bf404-4247-4e97-8c62-bec19bb76e2e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:16.290556 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290519    4274 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-kubeconfig\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.290556 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290555    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.290952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290579    4274 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-istio-csr-dns-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.290952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290593    4274 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/780bf404-4247-4e97-8c62-bec19bb76e2e-local-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.290952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290609    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2df\" (UniqueName: \"kubernetes.io/projected/780bf404-4247-4e97-8c62-bec19bb76e2e-kube-api-access-wj2df\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.290952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.290625    4274 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/780bf404-4247-4e97-8c62-bec19bb76e2e-cacerts\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:16.886929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.886898    4274 generic.go:356] "Generic (PLEG): container finished" podID="d704c925-67cb-480d-8130-2987c888b484" containerID="5e34ce68d27aad86bfe1ee3bfb922eafe9c85bd2672923499d9f16732f8fe12d" exitCode=60
Nov 03 21:15:16.887124 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.886973    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerDied","Data":"5e34ce68d27aad86bfe1ee3bfb922eafe9c85bd2672923499d9f16732f8fe12d"}
Nov 03 21:15:16.887124 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.887060    4274 scope.go:122] "RemoveContainer" containerID="9c3fa9125e3855a33ca8aa5756966419aba069510007777bd6d7c6f6df4abd91"
Nov 03 21:15:16.887446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.887413    4274 scope.go:122] "RemoveContainer" containerID="5e34ce68d27aad86bfe1ee3bfb922eafe9c85bd2672923499d9f16732f8fe12d"
Nov 03 21:15:16.887784 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:16.887751    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-swcj2_odh-ai-gateway-infra(d704c925-67cb-480d-8130-2987c888b484)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" podUID="d704c925-67cb-480d-8130-2987c888b484"
Nov 03 21:15:16.888717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.888694    4274 generic.go:356] "Generic (PLEG): container finished" podID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerID="e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21" exitCode=0
Nov 03 21:15:16.888821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.888727    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" event={"ID":"780bf404-4247-4e97-8c62-bec19bb76e2e","Type":"ContainerDied","Data":"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"}
Nov 03 21:15:16.888821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.888737    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"
Nov 03 21:15:16.888821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.888762    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4" event={"ID":"780bf404-4247-4e97-8c62-bec19bb76e2e","Type":"ContainerDied","Data":"0c171e2d3c370a1713783d986de04effd6dd9162cb6f764fe21e843ea54b5bca"}
Nov 03 21:15:16.899933 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.899912    4274 scope.go:122] "RemoveContainer" containerID="e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"
Nov 03 21:15:16.910169 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.910146    4274 scope.go:122] "RemoveContainer" containerID="e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"
Nov 03 21:15:16.910442 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:16.910421    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21\": container with ID starting with e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21 not found: ID does not exist" containerID="e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"
Nov 03 21:15:16.910510 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:16.910453    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21"} err="failed to get container status \"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21\": rpc error: code = NotFound desc = could not find container \"e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21\": container with ID starting with e13d9c5ff6c6463b1ef40a308c1199386c474acc1fc79a182df801448f140c21 not found: ID does not exist"
Nov 03 21:15:17.022697 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:17.022654    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:17.108296 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:17.108261    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 21:15:17.115110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:17.115074    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kmpq4"]
Nov 03 21:15:17.212603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:17.212531    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" path="/var/lib/kubelet/pods/780bf404-4247-4e97-8c62-bec19bb76e2e/volumes"
Nov 03 21:15:18.073528 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.073498    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:18.207279 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.207214    4274 scope.go:122] "RemoveContainer" containerID="e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628"
Nov 03 21:15:18.213172 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.213146    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\") pod \"d704c925-67cb-480d-8130-2987c888b484\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") "
Nov 03 21:15:18.213249 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.213234    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\" (UniqueName: \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\") pod \"d704c925-67cb-480d-8130-2987c888b484\" (UID: \"d704c925-67cb-480d-8130-2987c888b484\") "
Nov 03 21:15:18.213500 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.213480    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca" pod "d704c925-67cb-480d-8130-2987c888b484" (UID: "d704c925-67cb-480d-8130-2987c888b484"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:18.213843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.213822    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d704c925-67cb-480d-8130-2987c888b484-maas-api-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.215663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.215637    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj" pod "d704c925-67cb-480d-8130-2987c888b484" (UID: "d704c925-67cb-480d-8130-2987c888b484"). InnerVolumeSpecName "kube-api-access-9dndj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:18.314581 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.314551    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dndj\" (UniqueName: \"kubernetes.io/projected/d704c925-67cb-480d-8130-2987c888b484-kube-api-access-9dndj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.906332 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.906296    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerStarted","Data":"cc97a7e6d3e64d703b85fae923e514aca1a5354c5f83a45a61dc1660ab9601a8"}
Nov 03 21:15:18.907633 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.907602    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"
Nov 03 21:15:18.907775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.907692    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2" event={"ID":"d704c925-67cb-480d-8130-2987c888b484","Type":"ContainerDied","Data":"0f301fcbc798e723b03925d211b7220fd7423ce235bbd5ea040cae5499a09d78"}
Nov 03 21:15:18.907775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:18.907756    4274 scope.go:122] "RemoveContainer" containerID="5e34ce68d27aad86bfe1ee3bfb922eafe9c85bd2672923499d9f16732f8fe12d"
Nov 03 21:15:19.125396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.125365    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:19.129728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.129703    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-swcj2"]
Nov 03 21:15:19.214332 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.214260    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d704c925-67cb-480d-8130-2987c888b484" path="/var/lib/kubelet/pods/d704c925-67cb-480d-8130-2987c888b484/volumes"
Nov 03 21:15:19.917003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.916968    4274 generic.go:356] "Generic (PLEG): container finished" podID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerID="cc97a7e6d3e64d703b85fae923e514aca1a5354c5f83a45a61dc1660ab9601a8" exitCode=60
Nov 03 21:15:19.917208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.917071    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerDied","Data":"cc97a7e6d3e64d703b85fae923e514aca1a5354c5f83a45a61dc1660ab9601a8"}
Nov 03 21:15:19.917208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.917140    4274 scope.go:122] "RemoveContainer" containerID="e67ef158a496d8ee4fce948e3f03ab94a6d653716fe3cbd93ffb7e72cb95e628"
Nov 03 21:15:19.917529 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:19.917496    4274 scope.go:122] "RemoveContainer" containerID="cc97a7e6d3e64d703b85fae923e514aca1a5354c5f83a45a61dc1660ab9601a8"
Nov 03 21:15:19.917866 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:19.917840    4274 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk_odh-ai-gateway-infra(83f781bc-3dfa-45bf-b7ce-8d19954a8aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0"
Nov 03 21:15:20.034098 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.034056    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:20.845411 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.845380    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:15:20.846604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846564    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.846888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846605    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.846888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846691    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.846888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846699    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.847138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846948    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.847138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.846961    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.847138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847012    4274 cpu_manager.go:414] "containerMap: removing container" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery"
Nov 03 21:15:20.847138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847020    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery"
Nov 03 21:15:20.847455 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847420    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.847455 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847453    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="780bf404-4247-4e97-8c62-bec19bb76e2e" containerName="discovery"
Nov 03 21:15:20.847577 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847479    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.847577 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.847493    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d704c925-67cb-480d-8130-2987c888b484" containerName="cleanup"
Nov 03 21:15:20.854280 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.854253    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.906188 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.906161    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:15:20.946012 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.945984    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.946174 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.946024    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.946281 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.946252    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrs9\" (UniqueName: \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.946325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.946303    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.946446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.946420    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.946498 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.946469    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:20.948804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.948783    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c971dc\"/\"openshift-service-ca.crt\""
Nov 03 21:15:20.948804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.948799    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c971dc\"/\"default-dockercfg-tzt4f\""
Nov 03 21:15:20.948804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.948786    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c971dc\"/\"kube-root-ca.crt\""
Nov 03 21:15:20.949227 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:20.949209    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c971dc\"/\"test-model-c971dc-kserve-self-signed-certs\""
Nov 03 21:15:21.048182 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048139    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrs9\" (UniqueName: \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.048362 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048188    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.048362 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048239    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.048362 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048269    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.048519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048387    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.048519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.048413    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.049199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.049139    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.049499 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.049478    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.049598 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.049478    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.051853 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.050929    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.052307 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.052282    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.086911 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.086888    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:21.106140 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.106079    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrs9\" (UniqueName: \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\") pod \"test-model-c971dc-kserve-54d9c98797-f9d6v\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") " pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.167174 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.167149    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:21.251541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.251084    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\") pod \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") "
Nov 03 21:15:21.251541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.251119    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\" (UniqueName: \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\") pod \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\" (UID: \"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0\") "
Nov 03 21:15:21.251541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.251392    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca" pod "83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" (UID: "83f781bc-3dfa-45bf-b7ce-8d19954a8aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:21.251541 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.251506    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-maas-api-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:21.254414 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.254391    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg" pod "83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" (UID: "83f781bc-3dfa-45bf-b7ce-8d19954a8aa0"). InnerVolumeSpecName "kube-api-access-6qqsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:21.354189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.352999    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqsg\" (UniqueName: \"kubernetes.io/projected/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0-kube-api-access-6qqsg\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:21.544219 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.544189    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:15:21.545503 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:15:21.545462    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca408a92_57e6_4ec1_9faa_b1d735f4d88d.slice/crio-014eaf72519687633fc97efcd24e533ad2037ee46918b68c5a7b021e21999192 WatchSource:0}: Error finding container 014eaf72519687633fc97efcd24e533ad2037ee46918b68c5a7b021e21999192: Status 404 returned error can't find the container with id 014eaf72519687633fc97efcd24e533ad2037ee46918b68c5a7b021e21999192
Nov 03 21:15:21.935090 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.934974    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" event={"ID":"ca408a92-57e6-4ec1-9faa-b1d735f4d88d","Type":"ContainerStarted","Data":"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"}
Nov 03 21:15:21.935090 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.935019    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" event={"ID":"ca408a92-57e6-4ec1-9faa-b1d735f4d88d","Type":"ContainerStarted","Data":"014eaf72519687633fc97efcd24e533ad2037ee46918b68c5a7b021e21999192"}
Nov 03 21:15:21.936285 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.936261    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk" event={"ID":"83f781bc-3dfa-45bf-b7ce-8d19954a8aa0","Type":"ContainerDied","Data":"966a3f3efd4d31a21c23d7fe63c948449f4039abdd08d0b03e0f5469384f455a"}
Nov 03 21:15:21.936683 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.936276    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"
Nov 03 21:15:21.936683 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:21.936314    4274 scope.go:122] "RemoveContainer" containerID="cc97a7e6d3e64d703b85fae923e514aca1a5354c5f83a45a61dc1660ab9601a8"
Nov 03 21:15:22.046027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:22.045977    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" podStartSLOduration=2.045962921 podStartE2EDuration="2.045962921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:22.044418957 +0000 UTC m=+2746.970989205" watchObservedRunningTime="2026-11-03 21:15:22.045962921 +0000 UTC m=+2746.972533158"
Nov 03 21:15:22.151251 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:22.151223    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:22.158398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:22.158347    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cfcd54-29895675-szfhk"]
Nov 03 21:15:22.167767 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:22.167734    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:23.212761 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:23.212724    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" path="/var/lib/kubelet/pods/83f781bc-3dfa-45bf-b7ce-8d19954a8aa0/volumes"
Nov 03 21:15:27.366335 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:27.366302    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:15:31.168213 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:31.168177    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:31.180832 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:31.180806    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:32.001650 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:32.001618    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:15:56.306874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:56.306843    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 21:15:56.352909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:56.352875    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 21:15:56.355638 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:15:56.355585    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 21:15:57.366393 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:15:57.366358    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:16:03.447554 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.447520    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:16:03.448144 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.447700    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" podUID="b3559366-66db-42c9-87a2-27a543ad6816" containerName="maas-api" containerID="cri-o://1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e" gracePeriod=30
Nov 03 21:16:03.691472 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.691449    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:16:03.773947 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.773915    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\" (UniqueName: \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\") pod \"b3559366-66db-42c9-87a2-27a543ad6816\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") "
Nov 03 21:16:03.774200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.773970    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\") pod \"b3559366-66db-42c9-87a2-27a543ad6816\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") "
Nov 03 21:16:03.774200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.774009    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\") pod \"b3559366-66db-42c9-87a2-27a543ad6816\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") "
Nov 03 21:16:03.774200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.774123    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") pod \"b3559366-66db-42c9-87a2-27a543ad6816\" (UID: \"b3559366-66db-42c9-87a2-27a543ad6816\") "
Nov 03 21:16:03.774438 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.774416    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca" pod "b3559366-66db-42c9-87a2-27a543ad6816" (UID: "b3559366-66db-42c9-87a2-27a543ad6816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:03.774646 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.774620    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3559366-66db-42c9-87a2-27a543ad6816-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:03.776147 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.776123    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls" pod "b3559366-66db-42c9-87a2-27a543ad6816" (UID: "b3559366-66db-42c9-87a2-27a543ad6816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:03.776260 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.776184    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs" pod "b3559366-66db-42c9-87a2-27a543ad6816" (UID: "b3559366-66db-42c9-87a2-27a543ad6816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:03.776320 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.776259    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr" pod "b3559366-66db-42c9-87a2-27a543ad6816" (UID: "b3559366-66db-42c9-87a2-27a543ad6816"). InnerVolumeSpecName "kube-api-access-nndqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:03.875952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.875920    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:03.875952 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.875951    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndqr\" (UniqueName: \"kubernetes.io/projected/b3559366-66db-42c9-87a2-27a543ad6816-kube-api-access-nndqr\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:03.876153 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:03.875962    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3559366-66db-42c9-87a2-27a543ad6816-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:04.167101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.166985    4274 generic.go:356] "Generic (PLEG): container finished" podID="b3559366-66db-42c9-87a2-27a543ad6816" containerID="1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e" exitCode=0
Nov 03 21:16:04.167101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.167027    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"
Nov 03 21:16:04.167101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.167067    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" event={"ID":"b3559366-66db-42c9-87a2-27a543ad6816","Type":"ContainerDied","Data":"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"}
Nov 03 21:16:04.167363 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.167120    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x" event={"ID":"b3559366-66db-42c9-87a2-27a543ad6816","Type":"ContainerDied","Data":"1edcfb1acb1367f60baf719f1f4b1b8e77232b0ac610887f328f15bb0e0f6d69"}
Nov 03 21:16:04.167363 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.167144    4274 scope.go:122] "RemoveContainer" containerID="1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"
Nov 03 21:16:04.177415 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.177397    4274 scope.go:122] "RemoveContainer" containerID="1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"
Nov 03 21:16:04.177670 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:04.177653    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e\": container with ID starting with 1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e not found: ID does not exist" containerID="1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"
Nov 03 21:16:04.177737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.177682    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e"} err="failed to get container status \"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e\": rpc error: code = NotFound desc = could not find container \"1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e\": container with ID starting with 1a24b0d8f2adb7cbe19173488b276257e81dfe29cc57654b98359fb68265871e not found: ID does not exist"
Nov 03 21:16:04.320672 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.320638    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:16:04.324416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:04.324392    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfcd54-7474d4cb5-t8z2x"]
Nov 03 21:16:05.213070 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:05.213006    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3559366-66db-42c9-87a2-27a543ad6816" path="/var/lib/kubelet/pods/b3559366-66db-42c9-87a2-27a543ad6816/volumes"
Nov 03 21:16:11.327549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.327510    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main" containerID="cri-o://83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e" gracePeriod=45
Nov 03 21:16:11.373156 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.373121    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main" containerID="cri-o://1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42" gracePeriod=45
Nov 03 21:16:11.375081 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.375047    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" containerID="cri-o://560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25" gracePeriod=45
Nov 03 21:16:11.444889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.444845    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Nov 03 21:16:11.705148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.705123    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 21:16:11.708017 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.707998    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 21:16:11.711446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.711426    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 21:16:11.878918 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.878824    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.878918 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.878890    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.879199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.878927    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.879199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.878979    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.879199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879174    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.879370 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879173    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.879370 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879297    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.879370 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879327    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\" (UniqueName: \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.879559 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879379    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.879559 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879301    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.879559 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879430    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.879559 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879465    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\" (UniqueName: \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.879559 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879514    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.879821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879606    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\" (UniqueName: \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.879821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879792    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.880149 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.879693    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\" (UniqueName: \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.880467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.880402    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\" (UniqueName: \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\") pod \"248f67e5-66b7-4b53-9e09-86a28e073482\" (UID: \"248f67e5-66b7-4b53-9e09-86a28e073482\") "
Nov 03 21:16:11.880704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.880629    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.880704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.880674    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.881135 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881101    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.881373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881263    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.881373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881327    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.881617 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881573    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\") pod \"41ad8f9d-06ff-4fda-8b9f-506634484206\" (UID: \"41ad8f9d-06ff-4fda-8b9f-506634484206\") "
Nov 03 21:16:11.881697 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881621    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.882232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881807    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.882232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.881880    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\") pod \"1561b796-6182-4059-a706-9b53b22506b0\" (UID: \"1561b796-6182-4059-a706-9b53b22506b0\") "
Nov 03 21:16:11.882516 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.882486    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "kube-api-access-854kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.882905    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.882976    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.882990    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883004    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883019    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883047    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854kr\" (UniqueName: \"kubernetes.io/projected/41ad8f9d-06ff-4fda-8b9f-506634484206-kube-api-access-854kr\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883062    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883076    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883109 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883089    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.883754 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883709    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.883972 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.883939    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.884261 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.884229    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:11.884468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.884449    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:11.884583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.884527    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:11.884945 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.884917    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "kube-api-access-s54kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:11.885134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.885113    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm" pod "1561b796-6182-4059-a706-9b53b22506b0" (UID: "1561b796-6182-4059-a706-9b53b22506b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.885771 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.885740    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct" pod "248f67e5-66b7-4b53-9e09-86a28e073482" (UID: "248f67e5-66b7-4b53-9e09-86a28e073482"). InnerVolumeSpecName "kube-api-access-7blct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:11.886588 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.886563    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm" pod "41ad8f9d-06ff-4fda-8b9f-506634484206" (UID: "41ad8f9d-06ff-4fda-8b9f-506634484206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.984104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984071    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/248f67e5-66b7-4b53-9e09-86a28e073482-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984098    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984104 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984109    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/248f67e5-66b7-4b53-9e09-86a28e073482-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984118    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54kf\" (UniqueName: \"kubernetes.io/projected/1561b796-6182-4059-a706-9b53b22506b0-kube-api-access-s54kf\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984127    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ad8f9d-06ff-4fda-8b9f-506634484206-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984139    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blct\" (UniqueName: \"kubernetes.io/projected/248f67e5-66b7-4b53-9e09-86a28e073482-kube-api-access-7blct\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984148    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ad8f9d-06ff-4fda-8b9f-506634484206-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984156    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561b796-6182-4059-a706-9b53b22506b0-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:11.984425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:11.984164    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561b796-6182-4059-a706-9b53b22506b0-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.213198 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.213104    4274 generic.go:356] "Generic (PLEG): container finished" podID="248f67e5-66b7-4b53-9e09-86a28e073482" containerID="83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e" exitCode=0
Nov 03 21:16:12.213198 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.213169    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"
Nov 03 21:16:12.213440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.213192    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" event={"ID":"248f67e5-66b7-4b53-9e09-86a28e073482","Type":"ContainerDied","Data":"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"}
Nov 03 21:16:12.213440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.213240    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x" event={"ID":"248f67e5-66b7-4b53-9e09-86a28e073482","Type":"ContainerDied","Data":"1481016fea75cd2382cdb74a2819087d88f87b99e49ea4edc9c897289b76c7c3"}
Nov 03 21:16:12.213440 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.213263    4274 scope.go:122] "RemoveContainer" containerID="83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"
Nov 03 21:16:12.218676 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.218647    4274 generic.go:356] "Generic (PLEG): container finished" podID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerID="1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42" exitCode=0
Nov 03 21:16:12.218813 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.218698    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"
Nov 03 21:16:12.218813 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.218728    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" event={"ID":"41ad8f9d-06ff-4fda-8b9f-506634484206","Type":"ContainerDied","Data":"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"}
Nov 03 21:16:12.218813 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.218763    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk" event={"ID":"41ad8f9d-06ff-4fda-8b9f-506634484206","Type":"ContainerDied","Data":"0bdde147f5d1837adc4d836d03a9a65d13306c392545c5992ef4e89734daf6c4"}
Nov 03 21:16:12.220917 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.220888    4274 generic.go:356] "Generic (PLEG): container finished" podID="1561b796-6182-4059-a706-9b53b22506b0" containerID="560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25" exitCode=0
Nov 03 21:16:12.221010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.220923    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" event={"ID":"1561b796-6182-4059-a706-9b53b22506b0","Type":"ContainerDied","Data":"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"}
Nov 03 21:16:12.221010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.220950    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn" event={"ID":"1561b796-6182-4059-a706-9b53b22506b0","Type":"ContainerDied","Data":"839139347f7f8d9f640ea8a4a64b9e5da9708debbd13d9fa9feb4c3418decb62"}
Nov 03 21:16:12.221119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.221018    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"
Nov 03 21:16:12.228206 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.228180    4274 scope.go:122] "RemoveContainer" containerID="83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"
Nov 03 21:16:12.228497 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:12.228469    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e\": container with ID starting with 83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e not found: ID does not exist" containerID="83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"
Nov 03 21:16:12.228591 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.228500    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e"} err="failed to get container status \"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e\": rpc error: code = NotFound desc = could not find container \"83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e\": container with ID starting with 83f3572778bddf11de003867d40f62e658932a37d7d4814a288868cc45da166e not found: ID does not exist"
Nov 03 21:16:12.228591 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.228522    4274 scope.go:122] "RemoveContainer" containerID="1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"
Nov 03 21:16:12.236827 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.236812    4274 scope.go:122] "RemoveContainer" containerID="1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"
Nov 03 21:16:12.237124 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:12.237098    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42\": container with ID starting with 1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42 not found: ID does not exist" containerID="1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"
Nov 03 21:16:12.237177 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.237135    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42"} err="failed to get container status \"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42\": rpc error: code = NotFound desc = could not find container \"1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42\": container with ID starting with 1e5730199062a0469898f6a3f8f7d552d2ba77f69a82546280818f83ca51df42 not found: ID does not exist"
Nov 03 21:16:12.237177 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.237154    4274 scope.go:122] "RemoveContainer" containerID="560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"
Nov 03 21:16:12.244821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.244806    4274 scope.go:122] "RemoveContainer" containerID="560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"
Nov 03 21:16:12.245075 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:12.245056    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25\": container with ID starting with 560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25 not found: ID does not exist" containerID="560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"
Nov 03 21:16:12.245148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.245078    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25"} err="failed to get container status \"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25\": rpc error: code = NotFound desc = could not find container \"560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25\": container with ID starting with 560c1a8b6c0c10bd2ae3fddc057a18be752c6f1caa1ee11bbe9fe9a3f7605a25 not found: ID does not exist"
Nov 03 21:16:12.366268 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.366238    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 21:16:12.370795 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.370769    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-2-w2-3b8feb-kserve-76c54fc87b-khb4x"]
Nov 03 21:16:12.476622 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.476537    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 21:16:12.484263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.484234    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/distinct-w2-3b8feb-kserve-755948487-7vxvk"]
Nov 03 21:16:12.634337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.634302    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 21:16:12.640343 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:12.640318    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/premium-w2-3b8feb-kserve-f465d7b96-tz2kn"]
Nov 03 21:16:13.214506 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:13.214474    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1561b796-6182-4059-a706-9b53b22506b0" path="/var/lib/kubelet/pods/1561b796-6182-4059-a706-9b53b22506b0/volumes"
Nov 03 21:16:13.214900 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:13.214886    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" path="/var/lib/kubelet/pods/248f67e5-66b7-4b53-9e09-86a28e073482/volumes"
Nov 03 21:16:13.215311 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:13.215298    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" path="/var/lib/kubelet/pods/41ad8f9d-06ff-4fda-8b9f-506634484206/volumes"
Nov 03 21:16:19.049093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:19.049059    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 21:16:19.093320 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:19.093291    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 21:16:25.438873 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:25.438842    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:16:25.439320 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:25.439089    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" containerID="cri-o://24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c" gracePeriod=30
Nov 03 21:16:26.444325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:26.444284    4274 patch_prober.go:29] interesting pod/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:16:26.444767 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:26.444353    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:16:27.364972 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:27.364939    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:16:30.710639 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.710577    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:16:30.843658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843620    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.843845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843702    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.843845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843743    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.843845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843781    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.843845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843821    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.844081 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843863    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.844081 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843916    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.844081 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.843955    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.844240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844102    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\") pod \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\" (UID: \"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea\") "
Nov 03 21:16:30.844240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844135    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:30.844240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844213    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:30.844403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844263    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:30.844403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844364    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:30.844584 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844560    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:30.844663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844584    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.844663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844605    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.844663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844619    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.844663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.844635    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.845993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.845970    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:30.846152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.846133    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:16:30.846240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.846222    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:30.846282 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.846261    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz" pod "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" (UID: "d7d0a315-aaf8-4a5e-abf5-345fd81b5bea"). InnerVolumeSpecName "kube-api-access-q79mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:30.946232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.946197    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.946232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.946226    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.946232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.946236    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.946446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.946244    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q79mz\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-kube-api-access-q79mz\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:30.946446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:30.946254    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:31.334022 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.333979    4274 generic.go:356] "Generic (PLEG): container finished" podID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerID="24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c" exitCode=0
Nov 03 21:16:31.334233 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.334069    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerDied","Data":"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"}
Nov 03 21:16:31.334233 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.334093    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"
Nov 03 21:16:31.334233 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.334136    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g" event={"ID":"d7d0a315-aaf8-4a5e-abf5-345fd81b5bea","Type":"ContainerDied","Data":"44c0afc18d6238072d031eb14e25545f0839b31e6fd2b1d785af555bd177b123"}
Nov 03 21:16:31.334233 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.334157    4274 scope.go:122] "RemoveContainer" containerID="24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"
Nov 03 21:16:31.343806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.343787    4274 scope.go:122] "RemoveContainer" containerID="ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"
Nov 03 21:16:31.354638 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.354621    4274 scope.go:122] "RemoveContainer" containerID="24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"
Nov 03 21:16:31.354885 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:31.354866    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c\": container with ID starting with 24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c not found: ID does not exist" containerID="24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"
Nov 03 21:16:31.354950 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.354893    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c"} err="failed to get container status \"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c\": rpc error: code = NotFound desc = could not find container \"24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c\": container with ID starting with 24a763b27e9b12ae06eeb289de0998b8e6da778e96104349850f666902937e7c not found: ID does not exist"
Nov 03 21:16:31.354950 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.354915    4274 scope.go:122] "RemoveContainer" containerID="ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"
Nov 03 21:16:31.355185 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:31.355165    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922\": container with ID starting with ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922 not found: ID does not exist" containerID="ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"
Nov 03 21:16:31.355247 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.355202    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922"} err="failed to get container status \"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922\": rpc error: code = NotFound desc = could not find container \"ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922\": container with ID starting with ca93cb42b50827d7fcb8fcc5c28ed863466bd7fd64dbf15678864cfaed79d922 not found: ID does not exist"
Nov 03 21:16:31.489899 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.489796    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:16:31.536070 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:31.536013    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df71d7-openshift-default-687f69db89-4qx9g"]
Nov 03 21:16:33.212794 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:33.212763    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" path="/var/lib/kubelet/pods/d7d0a315-aaf8-4a5e-abf5-345fd81b5bea/volumes"
Nov 03 21:16:34.095251 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.095210    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main" containerID="cri-o://dc525f844ade43b7d4e3448ac38ae27b6b2e8cf0c6d0c7dc192821e3217f5802" gracePeriod=45
Nov 03 21:16:34.117347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.117302    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main" containerID="cri-o://91ca739c5612bbfaa80849d988c710f2bc795af7429a0a957c8deff354d192b7" gracePeriod=45
Nov 03 21:16:34.361630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.356502    4274 generic.go:356] "Generic (PLEG): container finished" podID="47623b5e-6801-4f45-9c81-6715852ab597" containerID="dc525f844ade43b7d4e3448ac38ae27b6b2e8cf0c6d0c7dc192821e3217f5802" exitCode=0
Nov 03 21:16:34.361630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.356641    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" event={"ID":"47623b5e-6801-4f45-9c81-6715852ab597","Type":"ContainerDied","Data":"dc525f844ade43b7d4e3448ac38ae27b6b2e8cf0c6d0c7dc192821e3217f5802"}
Nov 03 21:16:34.361630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.358690    4274 generic.go:356] "Generic (PLEG): container finished" podID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerID="91ca739c5612bbfaa80849d988c710f2bc795af7429a0a957c8deff354d192b7" exitCode=0
Nov 03 21:16:34.361630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.358832    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" event={"ID":"d4fadca4-3d39-460b-9ee4-01aff0c65391","Type":"ContainerDied","Data":"91ca739c5612bbfaa80849d988c710f2bc795af7429a0a957c8deff354d192b7"}
Nov 03 21:16:34.425755 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.425728    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 21:16:34.428887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.428865    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496519    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496576    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496610    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496669    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496717    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496762    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496789    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\" (UniqueName: \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496827    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496858    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496889    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\" (UniqueName: \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\") pod \"47623b5e-6801-4f45-9c81-6715852ab597\" (UID: \"47623b5e-6801-4f45-9c81-6715852ab597\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496908    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\" (UniqueName: \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.496936    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\") pod \"d4fadca4-3d39-460b-9ee4-01aff0c65391\" (UID: \"d4fadca4-3d39-460b-9ee4-01aff0c65391\") "
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.497620    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.497844    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.498031    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.498291    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.498305    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.498626    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.500870    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.500933    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.502345    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.502512    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "kube-api-access-dvmxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:34.505380 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.502581    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm" pod "d4fadca4-3d39-460b-9ee4-01aff0c65391" (UID: "d4fadca4-3d39-460b-9ee4-01aff0c65391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.507311 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.507282    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g" pod "47623b5e-6801-4f45-9c81-6715852ab597" (UID: "47623b5e-6801-4f45-9c81-6715852ab597"). InnerVolumeSpecName "kube-api-access-jgt4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:34.599431 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599372    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgt4g\" (UniqueName: \"kubernetes.io/projected/47623b5e-6801-4f45-9c81-6715852ab597-kube-api-access-jgt4g\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599431 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599433    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599450    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599488    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47623b5e-6801-4f45-9c81-6715852ab597-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599507    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmxp\" (UniqueName: \"kubernetes.io/projected/d4fadca4-3d39-460b-9ee4-01aff0c65391-kube-api-access-dvmxp\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599524    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599565    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599580    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4fadca4-3d39-460b-9ee4-01aff0c65391-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599595    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599610    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599647    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47623b5e-6801-4f45-9c81-6715852ab597-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.599802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:34.599660    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4fadca4-3d39-460b-9ee4-01aff0c65391-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:35.365171 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.365141    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"
Nov 03 21:16:35.365171 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.365165    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm" event={"ID":"d4fadca4-3d39-460b-9ee4-01aff0c65391","Type":"ContainerDied","Data":"b4b6f1f43769d9303ed842ca517df2feddb4dc19479b8e1a7d55e70ad126d309"}
Nov 03 21:16:35.365648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.365210    4274 scope.go:122] "RemoveContainer" containerID="91ca739c5612bbfaa80849d988c710f2bc795af7429a0a957c8deff354d192b7"
Nov 03 21:16:35.366612 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.366591    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6" event={"ID":"47623b5e-6801-4f45-9c81-6715852ab597","Type":"ContainerDied","Data":"ee4f246f05a97049d4f4a5368f29d15947263aacabe072b4d8640c1da49ea028"}
Nov 03 21:16:35.366694 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.366635    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"
Nov 03 21:16:35.373856 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.373840    4274 scope.go:122] "RemoveContainer" containerID="dc525f844ade43b7d4e3448ac38ae27b6b2e8cf0c6d0c7dc192821e3217f5802"
Nov 03 21:16:35.517524 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.517496    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 21:16:35.563650 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.563619    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-w3-88228a-kserve-55b5fbb57-zhdc6"]
Nov 03 21:16:35.623047 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.623020    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 21:16:35.628782 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:35.628757    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/simulator-w3-88228a-kserve-77fbcc8779-gfztm"]
Nov 03 21:16:37.213575 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:37.213538    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47623b5e-6801-4f45-9c81-6715852ab597" path="/var/lib/kubelet/pods/47623b5e-6801-4f45-9c81-6715852ab597/volumes"
Nov 03 21:16:37.213971 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:37.213955    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" path="/var/lib/kubelet/pods/d4fadca4-3d39-460b-9ee4-01aff0c65391/volumes"
Nov 03 21:16:42.848603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:42.848561    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 21:16:42.849116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:42.848825    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" podUID="d915e890-e4be-485e-b610-5dba66745dad" containerName="maas-api" containerID="cri-o://35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9" gracePeriod=30
Nov 03 21:16:43.121496 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.121468    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 21:16:43.197481 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197455    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\") pod \"d915e890-e4be-485e-b610-5dba66745dad\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") "
Nov 03 21:16:43.197602 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197532    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\" (UniqueName: \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\") pod \"d915e890-e4be-485e-b610-5dba66745dad\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") "
Nov 03 21:16:43.197602 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197552    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\") pod \"d915e890-e4be-485e-b610-5dba66745dad\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") "
Nov 03 21:16:43.197675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197609    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\") pod \"d915e890-e4be-485e-b610-5dba66745dad\" (UID: \"d915e890-e4be-485e-b610-5dba66745dad\") "
Nov 03 21:16:43.197813 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197790    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca" pod "d915e890-e4be-485e-b610-5dba66745dad" (UID: "d915e890-e4be-485e-b610-5dba66745dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:43.197905 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.197890    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d915e890-e4be-485e-b610-5dba66745dad-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:43.199722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.199696    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls" pod "d915e890-e4be-485e-b610-5dba66745dad" (UID: "d915e890-e4be-485e-b610-5dba66745dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:43.199722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.199704    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q" pod "d915e890-e4be-485e-b610-5dba66745dad" (UID: "d915e890-e4be-485e-b610-5dba66745dad"). InnerVolumeSpecName "kube-api-access-7f62q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:43.199867 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.199761    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs" pod "d915e890-e4be-485e-b610-5dba66745dad" (UID: "d915e890-e4be-485e-b610-5dba66745dad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:43.299131 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.299104    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f62q\" (UniqueName: \"kubernetes.io/projected/d915e890-e4be-485e-b610-5dba66745dad-kube-api-access-7f62q\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:43.299131 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.299130    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:43.299303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.299140    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d915e890-e4be-485e-b610-5dba66745dad-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:43.410354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.410259    4274 generic.go:356] "Generic (PLEG): container finished" podID="d915e890-e4be-485e-b610-5dba66745dad" containerID="35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9" exitCode=0
Nov 03 21:16:43.410354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.410311    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"
Nov 03 21:16:43.410354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.410339    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" event={"ID":"d915e890-e4be-485e-b610-5dba66745dad","Type":"ContainerDied","Data":"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"}
Nov 03 21:16:43.410657 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.410392    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn" event={"ID":"d915e890-e4be-485e-b610-5dba66745dad","Type":"ContainerDied","Data":"2b2c806e65564cd2cc0305360164355fa6a2c93bb9e1230a55c5a3c23f09d32e"}
Nov 03 21:16:43.410657 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.410415    4274 scope.go:122] "RemoveContainer" containerID="35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"
Nov 03 21:16:43.419876 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.419854    4274 scope.go:122] "RemoveContainer" containerID="35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"
Nov 03 21:16:43.420224 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:43.420202    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9\": container with ID starting with 35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9 not found: ID does not exist" containerID="35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"
Nov 03 21:16:43.420302 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.420234    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9"} err="failed to get container status \"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9\": rpc error: code = NotFound desc = could not find container \"35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9\": container with ID starting with 35f8341b186b693554c7e12f50ed0bbbd7ace44f93c25501112a1ce3da3643c9 not found: ID does not exist"
Nov 03 21:16:43.563526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.563494    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 21:16:43.567748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:43.567720    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3b8feb-57984cbc87-mp7gn"]
Nov 03 21:16:45.213622 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:16:45.213543    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d915e890-e4be-485e-b610-5dba66745dad" path="/var/lib/kubelet/pods/d915e890-e4be-485e-b610-5dba66745dad/volumes"
Nov 03 21:16:57.393273 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:16:57.393231    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:17:01.458488 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:01.458452    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:17:01.458889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:01.458656    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy" containerID="cri-o://2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186" gracePeriod=30
Nov 03 21:17:02.040624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:02.040580    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 21:17:02.040883 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:02.040846    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" containerID="cri-o://7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a" gracePeriod=30
Nov 03 21:17:02.857673 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:02.857636    4274 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:17:02.858096 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:02.857712    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:17:06.713175 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.713143    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:17:06.781728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781698    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.781929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781750    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.781929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781829    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.781929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781861    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.781929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781887    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.781929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781921    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.782237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781963    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.782237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.781997    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:06.782237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782055    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.782237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782096    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:06.782237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782141    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\") pod \"de804723-9f46-4882-b35d-e9e36a73880f\" (UID: \"de804723-9f46-4882-b35d-e9e36a73880f\") "
Nov 03 21:17:06.782493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782260    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:06.782493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782343    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:06.782493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782440    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:06.782703 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782678    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.782787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782704    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de804723-9f46-4882-b35d-e9e36a73880f-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.782787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782719    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.782787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782732    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.782787 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.782745    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.784283 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.784261    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:06.784382 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.784300    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:06.784568 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.784547    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:06.784612 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.784552    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth" pod "de804723-9f46-4882-b35d-e9e36a73880f" (UID: "de804723-9f46-4882-b35d-e9e36a73880f"). InnerVolumeSpecName "kube-api-access-2wkth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:06.883961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.883924    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de804723-9f46-4882-b35d-e9e36a73880f-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.883961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.883950    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.883961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.883961    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de804723-9f46-4882-b35d-e9e36a73880f-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:06.884221 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:06.883971    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wkth\" (UniqueName: \"kubernetes.io/projected/de804723-9f46-4882-b35d-e9e36a73880f-kube-api-access-2wkth\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.293537 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.293512    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 21:17:07.388761 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388693    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.388882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388777    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.388882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388834    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.388961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388881    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.388961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388918    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.389087 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388964    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.389087 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.388990    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:07.389200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389088    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.389200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389150    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.389200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389151    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:07.389359 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389197    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\") pod \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\" (UID: \"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f\") "
Nov 03 21:17:07.389359 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389227    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:07.389359 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389265    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:07.389583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389561    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.389658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389590    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.389658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389608    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.389658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389623    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.389828 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.389801    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:07.391362 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.391338    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:07.391580 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.391552    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:07.391580 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.391564    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:07.391717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.391602    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv" pod "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" (UID: "f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f"). InnerVolumeSpecName "kube-api-access-h9tmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:07.491199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.491169    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.491199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.491196    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.491325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.491205    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.491325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.491214    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.491325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.491223    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9tmv\" (UniqueName: \"kubernetes.io/projected/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f-kube-api-access-h9tmv\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:07.560993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.560956    4274 generic.go:356] "Generic (PLEG): container finished" podID="de804723-9f46-4882-b35d-e9e36a73880f" containerID="2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186" exitCode=0
Nov 03 21:17:07.561186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.561015    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"
Nov 03 21:17:07.561186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.561031    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" event={"ID":"de804723-9f46-4882-b35d-e9e36a73880f","Type":"ContainerDied","Data":"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"}
Nov 03 21:17:07.561186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.561098    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9" event={"ID":"de804723-9f46-4882-b35d-e9e36a73880f","Type":"ContainerDied","Data":"e30b8bde3ed53c3d43d1f1c9a6b686eeef0b30790f0155a07877720bcc0e7ac2"}
Nov 03 21:17:07.561186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.561129    4274 scope.go:122] "RemoveContainer" containerID="2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"
Nov 03 21:17:07.562483 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.562458    4274 generic.go:356] "Generic (PLEG): container finished" podID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerID="7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a" exitCode=0
Nov 03 21:17:07.562620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.562494    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" event={"ID":"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f","Type":"ContainerDied","Data":"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"}
Nov 03 21:17:07.562620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.562521    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"
Nov 03 21:17:07.562620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.562542    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p" event={"ID":"f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f","Type":"ContainerDied","Data":"be4f4ce7a6996b21889072ef207e582ad35d5f9e34c8e265b4f60c925815a10e"}
Nov 03 21:17:07.570635 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.570618    4274 scope.go:122] "RemoveContainer" containerID="2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"
Nov 03 21:17:07.570942 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:07.570870    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186\": container with ID starting with 2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186 not found: ID does not exist" containerID="2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"
Nov 03 21:17:07.570942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.570905    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186"} err="failed to get container status \"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186\": rpc error: code = NotFound desc = could not find container \"2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186\": container with ID starting with 2768953e44233c3abbf5cee22d78755020374511e0c3c43e02d51ca4f87a8186 not found: ID does not exist"
Nov 03 21:17:07.570942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.570928    4274 scope.go:122] "RemoveContainer" containerID="7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"
Nov 03 21:17:07.580259 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.580073    4274 scope.go:122] "RemoveContainer" containerID="7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"
Nov 03 21:17:07.580343 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:07.580323    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a\": container with ID starting with 7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a not found: ID does not exist" containerID="7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"
Nov 03 21:17:07.580395 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.580350    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a"} err="failed to get container status \"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a\": rpc error: code = NotFound desc = could not find container \"7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a\": container with ID starting with 7aa2369b6174ef993f9973f3703b9c60d037948cb755826edbbcafd05a0b884a not found: ID does not exist"
Nov 03 21:17:07.714869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.714789    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 21:17:07.719135 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.719106    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-bdl7p"]
Nov 03 21:17:07.820820 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.820787    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:17:07.834718 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:07.825826    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-v66n9"]
Nov 03 21:17:09.212444 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:09.212411    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de804723-9f46-4882-b35d-e9e36a73880f" path="/var/lib/kubelet/pods/de804723-9f46-4882-b35d-e9e36a73880f/volumes"
Nov 03 21:17:09.212872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:09.212857    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" path="/var/lib/kubelet/pods/f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f/volumes"
Nov 03 21:17:11.919747 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:11.919716    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 21:17:11.920199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:11.919900    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" podUID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" containerName="maas-api" containerID="cri-o://87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a" gracePeriod=30
Nov 03 21:17:12.172771 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.172710    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 21:17:12.243576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.243545    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\") pod \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") "
Nov 03 21:17:12.243721 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.243601    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\" (UniqueName: \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\") pod \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") "
Nov 03 21:17:12.243721 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.243626    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\") pod \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") "
Nov 03 21:17:12.243721 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.243647    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") pod \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\" (UID: \"78cc0ffc-1bf2-4901-81e6-9a80273f07c5\") "
Nov 03 21:17:12.244028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.244002    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca" pod "78cc0ffc-1bf2-4901-81e6-9a80273f07c5" (UID: "78cc0ffc-1bf2-4901-81e6-9a80273f07c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:12.245679 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.245654    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs" pod "78cc0ffc-1bf2-4901-81e6-9a80273f07c5" (UID: "78cc0ffc-1bf2-4901-81e6-9a80273f07c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:12.245825 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.245743    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls" pod "78cc0ffc-1bf2-4901-81e6-9a80273f07c5" (UID: "78cc0ffc-1bf2-4901-81e6-9a80273f07c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:12.245825 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.245758    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd" pod "78cc0ffc-1bf2-4901-81e6-9a80273f07c5" (UID: "78cc0ffc-1bf2-4901-81e6-9a80273f07c5"). InnerVolumeSpecName "kube-api-access-xzptd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:12.344671 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.344641    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:12.344671 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.344670    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzptd\" (UniqueName: \"kubernetes.io/projected/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-kube-api-access-xzptd\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:12.344857 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.344682    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:12.344857 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.344693    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cc0ffc-1bf2-4901-81e6-9a80273f07c5-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:12.589441 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.589407    4274 generic.go:356] "Generic (PLEG): container finished" podID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" containerID="87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a" exitCode=0
Nov 03 21:17:12.589655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.589458    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"
Nov 03 21:17:12.589655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.589496    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" event={"ID":"78cc0ffc-1bf2-4901-81e6-9a80273f07c5","Type":"ContainerDied","Data":"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"}
Nov 03 21:17:12.589655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.589551    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt" event={"ID":"78cc0ffc-1bf2-4901-81e6-9a80273f07c5","Type":"ContainerDied","Data":"8ccf363877fb71730dc25a79c4b57670ccb7414804eaed6d56463fd41953382b"}
Nov 03 21:17:12.589655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.589579    4274 scope.go:122] "RemoveContainer" containerID="87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"
Nov 03 21:17:12.599263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.599245    4274 scope.go:122] "RemoveContainer" containerID="87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"
Nov 03 21:17:12.599592 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:12.599560    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a\": container with ID starting with 87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a not found: ID does not exist" containerID="87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"
Nov 03 21:17:12.599702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.599600    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a"} err="failed to get container status \"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a\": rpc error: code = NotFound desc = could not find container \"87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a\": container with ID starting with 87e4fa8687d62fdde5cfa6a65248b6b581fe3c541c07c7a8a1956ed79f90a14a not found: ID does not exist"
Nov 03 21:17:12.742751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.742721    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 21:17:12.746598 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:12.746566    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88228a-98fdbc777-9w9xt"]
Nov 03 21:17:13.213446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:13.213417    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" path="/var/lib/kubelet/pods/78cc0ffc-1bf2-4901-81e6-9a80273f07c5/volumes"
Nov 03 21:17:15.171849 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:15.171818    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 21:17:27.365224 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:27.365183    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:17:30.191858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.191794    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" containerID="cri-o://2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247" gracePeriod=45
Nov 03 21:17:30.443031 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.442973    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 21:17:30.548761 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.548724    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\" (UniqueName: \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.548923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.548789    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.548923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.548859    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.549086 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.548932    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.549086 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.548967    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.549086 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549002    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\") pod \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\" (UID: \"09b80b7f-fbf8-44b4-9be5-035cb061d4e6\") "
Nov 03 21:17:30.549086 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549056    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:30.549291 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549230    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:30.549291 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549243    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:30.549383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549362    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.549383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549377    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.549486 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.549386    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.551274 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.551235    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:30.551437 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.551285    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "kube-api-access-jk8hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:30.551437 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.551350    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs" pod "09b80b7f-fbf8-44b4-9be5-035cb061d4e6" (UID: "09b80b7f-fbf8-44b4-9be5-035cb061d4e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:30.651077 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.651029    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.651077 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.651075    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8hx\" (UniqueName: \"kubernetes.io/projected/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-kube-api-access-jk8hx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.651266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.651090    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b80b7f-fbf8-44b4-9be5-035cb061d4e6-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:30.682657 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.682623    4274 generic.go:356] "Generic (PLEG): container finished" podID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerID="2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247" exitCode=0
Nov 03 21:17:30.682894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.682674    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"
Nov 03 21:17:30.682894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.682702    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" event={"ID":"09b80b7f-fbf8-44b4-9be5-035cb061d4e6","Type":"ContainerDied","Data":"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"}
Nov 03 21:17:30.682894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.682748    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" event={"ID":"09b80b7f-fbf8-44b4-9be5-035cb061d4e6","Type":"ContainerDied","Data":"ae223473c0d6530c875034aa061cbac94c84cecb0b7fa787283d623dde8b314a"}
Nov 03 21:17:30.682894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.682771    4274 scope.go:122] "RemoveContainer" containerID="2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"
Nov 03 21:17:30.692654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.692636    4274 scope.go:122] "RemoveContainer" containerID="2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"
Nov 03 21:17:30.692919 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:30.692898    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247\": container with ID starting with 2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247 not found: ID does not exist" containerID="2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"
Nov 03 21:17:30.692986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.692926    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247"} err="failed to get container status \"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247\": rpc error: code = NotFound desc = could not find container \"2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247\": container with ID starting with 2843fb55cd249812792e878454630b9497819e15d093a627a7930dc3f078f247 not found: ID does not exist"
Nov 03 21:17:30.839325 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.839286    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 21:17:30.845416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:30.845386    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq"]
Nov 03 21:17:31.213400 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:31.213326    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" path="/var/lib/kubelet/pods/09b80b7f-fbf8-44b4-9be5-035cb061d4e6/volumes"
Nov 03 21:17:31.381418 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:17:31.381383    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/premium-w1-0cd49b-kserve-5f7bff97b7-w77bq" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 21:17:57.365245 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:17:57.365208    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:18:13.170018 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:13.169982    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 21:18:13.170513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:13.170263    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy" containerID="cri-o://3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb" gracePeriod=30
Nov 03 21:18:18.419681 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.419654    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 21:18:18.528821 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.528797    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.528848    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.528895    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.528918    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.528955    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529025    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529075    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529116    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529123    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529156    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:18.529262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529168    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\") pod \"b0858a92-77a9-4d39-8836-a730a59bc83d\" (UID: \"b0858a92-77a9-4d39-8836-a730a59bc83d\") "
Nov 03 21:18:18.529842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529375    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:18.529842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529656    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.529842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529676    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.529842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529691    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.529842 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529726    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:18.530144 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.529961    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:18:18.531422 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.531389    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:18.531525 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.531501    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:18.531576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.531518    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:18:18.531653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.531634    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6" pod "b0858a92-77a9-4d39-8836-a730a59bc83d" (UID: "b0858a92-77a9-4d39-8836-a730a59bc83d"). InnerVolumeSpecName "kube-api-access-fbcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:18.630901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630872    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.630901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630897    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbcc6\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-kube-api-access-fbcc6\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.630901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630907    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.631110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630916    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0858a92-77a9-4d39-8836-a730a59bc83d-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.631110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630925    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0858a92-77a9-4d39-8836-a730a59bc83d-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.631110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.630934    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0858a92-77a9-4d39-8836-a730a59bc83d-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:18.919454 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.919368    4274 generic.go:356] "Generic (PLEG): container finished" podID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerID="3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb" exitCode=0
Nov 03 21:18:18.919454 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.919413    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"
Nov 03 21:18:18.919623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.919452    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" event={"ID":"b0858a92-77a9-4d39-8836-a730a59bc83d","Type":"ContainerDied","Data":"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"}
Nov 03 21:18:18.919623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.919501    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t" event={"ID":"b0858a92-77a9-4d39-8836-a730a59bc83d","Type":"ContainerDied","Data":"4beb734f90e2e23d03c1d70eb11f43710e97c94da99a55ab96dadc571b4cfd8b"}
Nov 03 21:18:18.919623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.919525    4274 scope.go:122] "RemoveContainer" containerID="3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"
Nov 03 21:18:18.929255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.929235    4274 scope.go:122] "RemoveContainer" containerID="3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"
Nov 03 21:18:18.929516 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:18:18.929499    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb\": container with ID starting with 3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb not found: ID does not exist" containerID="3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"
Nov 03 21:18:18.929562 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:18.929525    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb"} err="failed to get container status \"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb\": rpc error: code = NotFound desc = could not find container \"3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb\": container with ID starting with 3a27f7a7d8963f12e4ffe866b315b4b282bba4f5acd1b59f5ec82764c0f2a1fb not found: ID does not exist"
Nov 03 21:18:19.078171 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:19.078139    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 21:18:19.119451 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:19.119424    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-4x92t"]
Nov 03 21:18:19.215401 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:18:19.215331    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" path="/var/lib/kubelet/pods/b0858a92-77a9-4d39-8836-a730a59bc83d/volumes"
Nov 03 21:18:27.365477 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:18:27.365443    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:18:57.363792 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:18:57.363760    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:19:27.366003 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:19:27.365965    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:19:57.365295 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:19:57.365259    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:20:17.049483 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:17.049447    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:20:27.366153 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:20:27.366114    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:20:32.068906 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.068864    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" podUID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" containerName="main" containerID="cri-o://6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61" gracePeriod=45
Nov 03 21:20:32.329569 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.329511    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:20:32.449899 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.449873    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450083 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.449948    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\" (UniqueName: \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450083 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.449971    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450083 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.449993    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450083 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450014    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450327 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450145    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\") pod \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\" (UID: \"ca408a92-57e6-4ec1-9faa-b1d735f4d88d\") "
Nov 03 21:20:32.450327 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450304    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:32.450490 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450466    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:32.450604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450588    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.450673 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450610    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.450752 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.450735    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:32.452242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.452222    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:32.452324 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.452253    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:32.452538 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.452524    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9" pod "ca408a92-57e6-4ec1-9faa-b1d735f4d88d" (UID: "ca408a92-57e6-4ec1-9faa-b1d735f4d88d"). InnerVolumeSpecName "kube-api-access-xfrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:32.551401 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.551368    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.551401 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.551396    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.551586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.551410    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.551586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.551423    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrs9\" (UniqueName: \"kubernetes.io/projected/ca408a92-57e6-4ec1-9faa-b1d735f4d88d-kube-api-access-xfrs9\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:32.595639 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.595572    4274 generic.go:356] "Generic (PLEG): container finished" podID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" containerID="6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61" exitCode=0
Nov 03 21:20:32.595639 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.595620    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"
Nov 03 21:20:32.595878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.595658    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" event={"ID":"ca408a92-57e6-4ec1-9faa-b1d735f4d88d","Type":"ContainerDied","Data":"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"}
Nov 03 21:20:32.595878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.595706    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v" event={"ID":"ca408a92-57e6-4ec1-9faa-b1d735f4d88d","Type":"ContainerDied","Data":"014eaf72519687633fc97efcd24e533ad2037ee46918b68c5a7b021e21999192"}
Nov 03 21:20:32.595878 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.595725    4274 scope.go:122] "RemoveContainer" containerID="6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"
Nov 03 21:20:32.604727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.604710    4274 scope.go:122] "RemoveContainer" containerID="6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"
Nov 03 21:20:32.604990 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:20:32.604974    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61\": container with ID starting with 6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61 not found: ID does not exist" containerID="6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"
Nov 03 21:20:32.605048 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.604997    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61"} err="failed to get container status \"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61\": rpc error: code = NotFound desc = could not find container \"6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61\": container with ID starting with 6df259a27601b2413540253274bda947a2466a0be528856810ebdc23e080fa61 not found: ID does not exist"
Nov 03 21:20:32.749191 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.749159    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:20:32.753055 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:32.753013    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c971dc/test-model-c971dc-kserve-54d9c98797-f9d6v"]
Nov 03 21:20:33.212505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:33.212473    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" path="/var/lib/kubelet/pods/ca408a92-57e6-4ec1-9faa-b1d735f4d88d/volumes"
Nov 03 21:20:43.779145 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:43.779114    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:20:43.779574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:43.779345    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy" containerID="cri-o://04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127" gracePeriod=30
Nov 03 21:20:49.055967 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.055904    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:20:49.104849 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.104815    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.104849 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.104853    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.104969    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105001    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105119 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105069    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105287 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105127    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105287 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105188    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105287 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105222    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105287 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105250    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\") pod \"c74b724f-1f77-41d3-beef-430899abeb4d\" (UID: \"c74b724f-1f77-41d3-beef-430899abeb4d\") "
Nov 03 21:20:49.105491 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105270    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.105491 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105352    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.105596 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105489    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.105766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105719    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.105766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105745    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.105894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105796    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.105894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.105824    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.106182 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.106125    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:49.107475 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.107450    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "kube-api-access-jt5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:49.107626 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.107603    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.107675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.107635    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:49.108025 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.108002    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo" pod "c74b724f-1f77-41d3-beef-430899abeb4d" (UID: "c74b724f-1f77-41d3-beef-430899abeb4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:20:49.207148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207108    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.207148 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207147    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74b724f-1f77-41d3-beef-430899abeb4d-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.207385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207163    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5ps\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-kube-api-access-jt5ps\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.207385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207178    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74b724f-1f77-41d3-beef-430899abeb4d-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.207385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207192    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74b724f-1f77-41d3-beef-430899abeb4d-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.207385 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.207282    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74b724f-1f77-41d3-beef-430899abeb4d-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.684047 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.684010    4274 generic.go:356] "Generic (PLEG): container finished" podID="c74b724f-1f77-41d3-beef-430899abeb4d" containerID="04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127" exitCode=0
Nov 03 21:20:49.684254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.684071    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" event={"ID":"c74b724f-1f77-41d3-beef-430899abeb4d","Type":"ContainerDied","Data":"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"}
Nov 03 21:20:49.684254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.684139    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg" event={"ID":"c74b724f-1f77-41d3-beef-430899abeb4d","Type":"ContainerDied","Data":"92a8842818ab3dc8e3a9d9af0da4b957c4fd4e24797ff4ebf131281bc9fc3b7c"}
Nov 03 21:20:49.684254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.684082    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"
Nov 03 21:20:49.684254 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.684171    4274 scope.go:122] "RemoveContainer" containerID="04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"
Nov 03 21:20:49.693060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.693020    4274 scope.go:122] "RemoveContainer" containerID="04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"
Nov 03 21:20:49.693347 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:20:49.693323    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127\": container with ID starting with 04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127 not found: ID does not exist" containerID="04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"
Nov 03 21:20:49.693392 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.693353    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127"} err="failed to get container status \"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127\": rpc error: code = NotFound desc = could not find container \"04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127\": container with ID starting with 04d6d981da8ec279c854293b931ecc21dd049add7497f65db02f48fe7814f127 not found: ID does not exist"
Nov 03 21:20:49.839345 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.839314    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:20:49.885178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:49.885151    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-8zlpg"]
Nov 03 21:20:51.212731 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:51.212700    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" path="/var/lib/kubelet/pods/c74b724f-1f77-41d3-beef-430899abeb4d/volumes"
Nov 03 21:20:56.207323 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:56.207290    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:20:56.319152 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:20:56.319121    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:20:57.367272 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:20:57.367235    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:21:01.319732 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:01.319673    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing" containerID="cri-o://c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f" gracePeriod=25
Nov 03 21:21:01.720354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:01.720259    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing" containerID="cri-o://88fe25b38fcbc1298fbfe1add06a5d95bf89f3135da5ed384709cebd6da0e59a" gracePeriod=25
Nov 03 21:21:10.731597 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:10.731558    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:10.731597 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:10.731589    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:20.731461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:20.731381    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:20.731839 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:20.731381    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:21.571845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.571821    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:21:21.712557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.712481    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\" (UniqueName: \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\") pod \"95a39c18-5bcf-4ddf-b436-9b507c712d74\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") "
Nov 03 21:21:21.712695 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.712589    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\") pod \"95a39c18-5bcf-4ddf-b436-9b507c712d74\" (UID: \"95a39c18-5bcf-4ddf-b436-9b507c712d74\") "
Nov 03 21:21:21.712920 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.712894    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume" pod "95a39c18-5bcf-4ddf-b436-9b507c712d74" (UID: "95a39c18-5bcf-4ddf-b436-9b507c712d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:21.713069 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.713022    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a39c18-5bcf-4ddf-b436-9b507c712d74-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:21.714513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.714487    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w" pod "95a39c18-5bcf-4ddf-b436-9b507c712d74" (UID: "95a39c18-5bcf-4ddf-b436-9b507c712d74"). InnerVolumeSpecName "kube-api-access-g444w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:21.814203 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.814179    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g444w\" (UniqueName: \"kubernetes.io/projected/95a39c18-5bcf-4ddf-b436-9b507c712d74-kube-api-access-g444w\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:21.837215 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.837187    4274 generic.go:356] "Generic (PLEG): container finished" podID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerID="c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f" exitCode=0
Nov 03 21:21:21.837340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.837234    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"
Nov 03 21:21:21.837340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.837271    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" event={"ID":"95a39c18-5bcf-4ddf-b436-9b507c712d74","Type":"ContainerDied","Data":"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"}
Nov 03 21:21:21.837340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.837336    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6" event={"ID":"95a39c18-5bcf-4ddf-b436-9b507c712d74","Type":"ContainerDied","Data":"629ba6d93346e6534c37d6d4fc8ebf4c1be91806ad482682bd0b741dde8ab28c"}
Nov 03 21:21:21.837587 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.837360    4274 scope.go:122] "RemoveContainer" containerID="c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"
Nov 03 21:21:21.839247 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.839222    4274 generic.go:356] "Generic (PLEG): container finished" podID="a3c871ba-1376-444f-9242-0e719398a2c0" containerID="88fe25b38fcbc1298fbfe1add06a5d95bf89f3135da5ed384709cebd6da0e59a" exitCode=0
Nov 03 21:21:21.839340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.839270    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" event={"ID":"a3c871ba-1376-444f-9242-0e719398a2c0","Type":"ContainerDied","Data":"88fe25b38fcbc1298fbfe1add06a5d95bf89f3135da5ed384709cebd6da0e59a"}
Nov 03 21:21:21.846321 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.846305    4274 scope.go:122] "RemoveContainer" containerID="c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"
Nov 03 21:21:21.846536 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:21.846516    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f\": container with ID starting with c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f not found: ID does not exist" containerID="c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"
Nov 03 21:21:21.846589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.846543    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f"} err="failed to get container status \"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f\": rpc error: code = NotFound desc = could not find container \"c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f\": container with ID starting with c75ad355c79215da92a5a95bbf9c8526e61e8f2b3ae22ef7c1376918c0dc5a7f not found: ID does not exist"
Nov 03 21:21:21.952957 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.952926    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:21:21.992329 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.992273    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:21:21.996454 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:21.996433    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c971dc-66858fd8c5-5g5t6"]
Nov 03 21:21:22.117162 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.117130    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\") pod \"a3c871ba-1376-444f-9242-0e719398a2c0\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") "
Nov 03 21:21:22.117319 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.117279    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\" (UniqueName: \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\") pod \"a3c871ba-1376-444f-9242-0e719398a2c0\" (UID: \"a3c871ba-1376-444f-9242-0e719398a2c0\") "
Nov 03 21:21:22.117493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.117472    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume" pod "a3c871ba-1376-444f-9242-0e719398a2c0" (UID: "a3c871ba-1376-444f-9242-0e719398a2c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:22.119312 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.119284    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9" pod "a3c871ba-1376-444f-9242-0e719398a2c0" (UID: "a3c871ba-1376-444f-9242-0e719398a2c0"). InnerVolumeSpecName "kube-api-access-lk6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:22.218768 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.218733    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk6w9\" (UniqueName: \"kubernetes.io/projected/a3c871ba-1376-444f-9242-0e719398a2c0-kube-api-access-lk6w9\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:22.218768 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.218759    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3c871ba-1376-444f-9242-0e719398a2c0-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:22.846102 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.846062    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89" event={"ID":"a3c871ba-1376-444f-9242-0e719398a2c0","Type":"ContainerDied","Data":"3a481028c59a8902eec37840d3d7a377fb984f96e1b5ed830079e50afdaac296"}
Nov 03 21:21:22.846526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.846101    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"
Nov 03 21:21:22.846526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.846151    4274 scope.go:122] "RemoveContainer" containerID="88fe25b38fcbc1298fbfe1add06a5d95bf89f3135da5ed384709cebd6da0e59a"
Nov 03 21:21:22.999558 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:22.999526    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:21:23.045742 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:23.045711    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c971dc-5ccf66dd69-fqw89"]
Nov 03 21:21:23.212630 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:23.212558    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" path="/var/lib/kubelet/pods/95a39c18-5bcf-4ddf-b436-9b507c712d74/volumes"
Nov 03 21:21:23.212950 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:23.212936    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" path="/var/lib/kubelet/pods/a3c871ba-1376-444f-9242-0e719398a2c0/volumes"
Nov 03 21:21:25.344351 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:25.344318    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:21:25.344802 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:25.344514    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy" containerID="cri-o://862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa" gracePeriod=30
Nov 03 21:21:27.365092 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:27.365046    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:21:30.592524 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.592490    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:21:30.593576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593555    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593590    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593613    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593675 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593648    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593700    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593755    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593799    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.593843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593830    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.594065 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593890    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:30.594065 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593916    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\") pod \"cbcae569-67ea-4a3b-95e6-085806de6f58\" (UID: \"cbcae569-67ea-4a3b-95e6-085806de6f58\") "
Nov 03 21:21:30.594065 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.593939    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:30.594065 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594027    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:30.594298 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594203    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:30.594369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594294    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.594369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594314    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.594369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594328    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.594369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594343    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.594583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.594373    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:30.595801 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.595780    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:21:30.595910 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.595808    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "kube-api-access-6hbmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:30.595910 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.595885    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:30.595996 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.595970    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token" pod "cbcae569-67ea-4a3b-95e6-085806de6f58" (UID: "cbcae569-67ea-4a3b-95e6-085806de6f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:30.694942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.694859    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.694942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.694897    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.694942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.694906    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbcae569-67ea-4a3b-95e6-085806de6f58-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.694942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.694915    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbcae569-67ea-4a3b-95e6-085806de6f58-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.694942 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.694924    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hbmm\" (UniqueName: \"kubernetes.io/projected/cbcae569-67ea-4a3b-95e6-085806de6f58-kube-api-access-6hbmm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:30.810031 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.809999    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 21:21:30.886921 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.886883    4274 generic.go:356] "Generic (PLEG): container finished" podID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerID="862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa" exitCode=0
Nov 03 21:21:30.887136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.886943    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"
Nov 03 21:21:30.887136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.886962    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerDied","Data":"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"}
Nov 03 21:21:30.887136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.887016    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92" event={"ID":"cbcae569-67ea-4a3b-95e6-085806de6f58","Type":"ContainerDied","Data":"d36a1c0136c06a5bc94375656e483c5ce107b0fb5256899d074f7e2406f0462d"}
Nov 03 21:21:30.887136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.887053    4274 scope.go:122] "RemoveContainer" containerID="862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"
Nov 03 21:21:30.897236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.897216    4274 scope.go:122] "RemoveContainer" containerID="27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"
Nov 03 21:21:30.905053 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.905017    4274 scope.go:122] "RemoveContainer" containerID="862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"
Nov 03 21:21:30.905311 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:30.905292    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa\": container with ID starting with 862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa not found: ID does not exist" containerID="862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"
Nov 03 21:21:30.905416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.905319    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa"} err="failed to get container status \"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa\": rpc error: code = NotFound desc = could not find container \"862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa\": container with ID starting with 862dc689c8032d250e3e125c26e6f42e44363e54eeb374b76795eef3354d1dfa not found: ID does not exist"
Nov 03 21:21:30.905416 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.905336    4274 scope.go:122] "RemoveContainer" containerID="27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"
Nov 03 21:21:30.905539 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:30.905525    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e\": container with ID starting with 27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e not found: ID does not exist" containerID="27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"
Nov 03 21:21:30.905581 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:30.905543    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e"} err="failed to get container status \"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e\": rpc error: code = NotFound desc = could not find container \"27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e\": container with ID starting with 27661f2c1c036ff39ffb47d513f8e13cbdef4cca46855fda1a21575b6548399e not found: ID does not exist"
Nov 03 21:21:31.030126 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:31.030090    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:21:31.033572 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:31.033547    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c971dc-openshift-default-7768f69859-qbb92"]
Nov 03 21:21:31.211899 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:31.211863    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" path="/var/lib/kubelet/pods/cbcae569-67ea-4a3b-95e6-085806de6f58/volumes"
Nov 03 21:21:38.015363 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:38.015326    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 21:21:43.016275 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:43.016223    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing" containerID="cri-o://45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17" gracePeriod=25
Nov 03 21:21:43.724430 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:43.724383    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 21:21:43.724640 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:43.724603    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy" containerID="cri-o://a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3" gracePeriod=30
Nov 03 21:21:45.830120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:45.830051    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main" containerID="cri-o://eab9a9e78dc01a11f11f5bfc95844a621a2c61759b8f65d89ead8ef36c1c4e64" gracePeriod=45
Nov 03 21:21:45.960652 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:45.960621    4274 generic.go:356] "Generic (PLEG): container finished" podID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerID="eab9a9e78dc01a11f11f5bfc95844a621a2c61759b8f65d89ead8ef36c1c4e64" exitCode=0
Nov 03 21:21:45.960811 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:45.960694    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" event={"ID":"3627eb01-f4b2-4012-8cc9-a6f0383bc610","Type":"ContainerDied","Data":"eab9a9e78dc01a11f11f5bfc95844a621a2c61759b8f65d89ead8ef36c1c4e64"}
Nov 03 21:21:46.085299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.085239    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 21:21:46.132638 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132614    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\" (UniqueName: \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.132779 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132691    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\" (UniqueName: \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.132779 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132765    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.132910 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132895    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.132962 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132931    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.133025 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.132992    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\") pod \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\" (UID: \"3627eb01-f4b2-4012-8cc9-a6f0383bc610\") "
Nov 03 21:21:46.133099 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.133057    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.133257 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.133232    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.133334 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.133316    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.133452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.133390    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.135101 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.135075    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "kube-api-access-rz7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:46.135212 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.135156    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:46.135349 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.135329    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm" pod "3627eb01-f4b2-4012-8cc9-a6f0383bc610" (UID: "3627eb01-f4b2-4012-8cc9-a6f0383bc610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.234778 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.234739    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.234929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.234782    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7z7\" (UniqueName: \"kubernetes.io/projected/3627eb01-f4b2-4012-8cc9-a6f0383bc610-kube-api-access-rz7z7\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.234929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.234799    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3627eb01-f4b2-4012-8cc9-a6f0383bc610-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.234929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.234812    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.234929 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.234826    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3627eb01-f4b2-4012-8cc9-a6f0383bc610-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.967090 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.967060    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"
Nov 03 21:21:46.967522 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.967089    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s" event={"ID":"3627eb01-f4b2-4012-8cc9-a6f0383bc610","Type":"ContainerDied","Data":"04d911cfce20483c2d4ed50eb6bfb0fb9ffbb552a2e090be9754ab9204715e95"}
Nov 03 21:21:46.967522 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:46.967147    4274 scope.go:122] "RemoveContainer" containerID="eab9a9e78dc01a11f11f5bfc95844a621a2c61759b8f65d89ead8ef36c1c4e64"
Nov 03 21:21:47.127458 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:47.127429    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 21:21:47.132775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:47.132752    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/simulator-w6-504eb0-kserve-545988648f-tt95s"]
Nov 03 21:21:47.176610 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:47.176577    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:47.214682 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:47.214652    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" path="/var/lib/kubelet/pods/3627eb01-f4b2-4012-8cc9-a6f0383bc610/volumes"
Nov 03 21:21:48.970200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.970171    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 21:21:48.977571 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.977547    4274 generic.go:356] "Generic (PLEG): container finished" podID="2689076e-c8a2-49fb-8302-6576a600ba29" containerID="a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3" exitCode=0
Nov 03 21:21:48.977701 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.977588    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"
Nov 03 21:21:48.977701 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.977587    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" event={"ID":"2689076e-c8a2-49fb-8302-6576a600ba29","Type":"ContainerDied","Data":"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"}
Nov 03 21:21:48.977791 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.977710    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5" event={"ID":"2689076e-c8a2-49fb-8302-6576a600ba29","Type":"ContainerDied","Data":"92025ad0d873f0cc38867c846ee50b6541b6d86d9791663c0631e0185d4d3f25"}
Nov 03 21:21:48.977791 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.977733    4274 scope.go:122] "RemoveContainer" containerID="a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"
Nov 03 21:21:48.986113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.986094    4274 scope.go:122] "RemoveContainer" containerID="a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"
Nov 03 21:21:48.986368 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:48.986349    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3\": container with ID starting with a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3 not found: ID does not exist" containerID="a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"
Nov 03 21:21:48.986408 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:48.986377    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3"} err="failed to get container status \"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3\": rpc error: code = NotFound desc = could not find container \"a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3\": container with ID starting with a61b9590c1da7dded37fd1028ba525614e362a8e90a92325e2f7f257510003e3 not found: ID does not exist"
Nov 03 21:21:49.061302 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061275    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061331    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061358    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061385    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061427 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061404    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061433    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061558    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:49.061605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061568    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061629    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061701    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:49.061757 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061721    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\") pod \"2689076e-c8a2-49fb-8302-6576a600ba29\" (UID: \"2689076e-c8a2-49fb-8302-6576a600ba29\") "
Nov 03 21:21:49.061900 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061810    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:49.061951 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.061932    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:49.062120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.062094    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:49.062236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.062143    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.062236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.062163    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2689076e-c8a2-49fb-8302-6576a600ba29-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.062236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.062178    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.062236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.062192    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.063632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.063594    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "kube-api-access-lcl6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:49.063632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.063615    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:49.063762 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.063669    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:21:49.063762 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.063738    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token" pod "2689076e-c8a2-49fb-8302-6576a600ba29" (UID: "2689076e-c8a2-49fb-8302-6576a600ba29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:49.163523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.163493    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.163523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.163518    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2689076e-c8a2-49fb-8302-6576a600ba29-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.163523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.163527    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl6l\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-kube-api-access-lcl6l\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.163724 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.163537    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2689076e-c8a2-49fb-8302-6576a600ba29-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.163724 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.163545    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2689076e-c8a2-49fb-8302-6576a600ba29-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:49.232700 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:49.232668    4274 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2689076e_c8a2_49fb_8302_6576a600ba29.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 21:21:49.385563 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.385480    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 21:21:49.390702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:49.390675    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-v24v5"]
Nov 03 21:21:51.212312 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:51.212280    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" path="/var/lib/kubelet/pods/2689076e-c8a2-49fb-8302-6576a600ba29/volumes"
Nov 03 21:21:53.491931 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.491883    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 21:21:53.492475 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.492157    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" podUID="31955084-06da-41d1-9adb-5a44bc622645" containerName="maas-api" containerID="cri-o://30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de" gracePeriod=30
Nov 03 21:21:53.750628 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.750570    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 21:21:53.809609 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.809579    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") pod \"31955084-06da-41d1-9adb-5a44bc622645\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") "
Nov 03 21:21:53.809760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.809616    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\") pod \"31955084-06da-41d1-9adb-5a44bc622645\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") "
Nov 03 21:21:53.809760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.809663    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\") pod \"31955084-06da-41d1-9adb-5a44bc622645\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") "
Nov 03 21:21:53.809760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.809725    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\" (UniqueName: \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\") pod \"31955084-06da-41d1-9adb-5a44bc622645\" (UID: \"31955084-06da-41d1-9adb-5a44bc622645\") "
Nov 03 21:21:53.810063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.810020    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca" pod "31955084-06da-41d1-9adb-5a44bc622645" (UID: "31955084-06da-41d1-9adb-5a44bc622645"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:53.811665 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.811644    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls" pod "31955084-06da-41d1-9adb-5a44bc622645" (UID: "31955084-06da-41d1-9adb-5a44bc622645"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:53.812228 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.812199    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb" pod "31955084-06da-41d1-9adb-5a44bc622645" (UID: "31955084-06da-41d1-9adb-5a44bc622645"). InnerVolumeSpecName "kube-api-access-l4lnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:53.812228 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.812209    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs" pod "31955084-06da-41d1-9adb-5a44bc622645" (UID: "31955084-06da-41d1-9adb-5a44bc622645"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:53.911505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.911472    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lnb\" (UniqueName: \"kubernetes.io/projected/31955084-06da-41d1-9adb-5a44bc622645-kube-api-access-l4lnb\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:53.911505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.911497    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:53.911505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.911507    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31955084-06da-41d1-9adb-5a44bc622645-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:53.911732 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:53.911518    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31955084-06da-41d1-9adb-5a44bc622645-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:54.002442 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.002354    4274 generic.go:356] "Generic (PLEG): container finished" podID="31955084-06da-41d1-9adb-5a44bc622645" containerID="30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de" exitCode=0
Nov 03 21:21:54.002442 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.002402    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"
Nov 03 21:21:54.002640 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.002433    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" event={"ID":"31955084-06da-41d1-9adb-5a44bc622645","Type":"ContainerDied","Data":"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"}
Nov 03 21:21:54.002640 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.002488    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7" event={"ID":"31955084-06da-41d1-9adb-5a44bc622645","Type":"ContainerDied","Data":"95374b3ea9eec29cad199be96f541e1f29bfd18fcdf64312f2a133468a9500ea"}
Nov 03 21:21:54.002640 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.002511    4274 scope.go:122] "RemoveContainer" containerID="30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"
Nov 03 21:21:54.012726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.012704    4274 scope.go:122] "RemoveContainer" containerID="30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"
Nov 03 21:21:54.013011 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:54.012979    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de\": container with ID starting with 30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de not found: ID does not exist" containerID="30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"
Nov 03 21:21:54.013147 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.013014    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de"} err="failed to get container status \"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de\": rpc error: code = NotFound desc = could not find container \"30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de\": container with ID starting with 30408ec414580751cd3789ee9056ee35f2afc54f30963f0fd1b621dbf88738de not found: ID does not exist"
Nov 03 21:21:54.156519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.156485    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 21:21:54.160518 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:54.160493    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-504eb0-5f59678467-fg8c7"]
Nov 03 21:21:55.211642 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:55.211600    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31955084-06da-41d1-9adb-5a44bc622645" path="/var/lib/kubelet/pods/31955084-06da-41d1-9adb-5a44bc622645/volumes"
Nov 03 21:21:57.175888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:21:57.175850    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:57.364669 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:21:57.364630    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:22:03.061781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.061750    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 21:22:03.062184 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.061970    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy" containerID="cri-o://2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016" gracePeriod=30
Nov 03 21:22:03.257649 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.257621    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 21:22:03.394255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.394180    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\") pod \"b891c86c-18e7-4a20-9d8f-af05861c1599\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") "
Nov 03 21:22:03.394394 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.394326    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\" (UniqueName: \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\") pod \"b891c86c-18e7-4a20-9d8f-af05861c1599\" (UID: \"b891c86c-18e7-4a20-9d8f-af05861c1599\") "
Nov 03 21:22:03.394651 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.394624    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume" pod "b891c86c-18e7-4a20-9d8f-af05861c1599" (UID: "b891c86c-18e7-4a20-9d8f-af05861c1599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:03.396396 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.396366    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz" pod "b891c86c-18e7-4a20-9d8f-af05861c1599" (UID: "b891c86c-18e7-4a20-9d8f-af05861c1599"). InnerVolumeSpecName "kube-api-access-pgppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:03.495752 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.495710    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgppz\" (UniqueName: \"kubernetes.io/projected/b891c86c-18e7-4a20-9d8f-af05861c1599-kube-api-access-pgppz\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:03.495752 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:03.495744    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b891c86c-18e7-4a20-9d8f-af05861c1599-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:04.050095 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.050057    4274 generic.go:356] "Generic (PLEG): container finished" podID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerID="45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17" exitCode=0
Nov 03 21:22:04.050299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.050124    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" event={"ID":"b891c86c-18e7-4a20-9d8f-af05861c1599","Type":"ContainerDied","Data":"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"}
Nov 03 21:22:04.050299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.050160    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"
Nov 03 21:22:04.050299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.050179    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf" event={"ID":"b891c86c-18e7-4a20-9d8f-af05861c1599","Type":"ContainerDied","Data":"476b65bbdfa54f176965a7c90fb89372a5e4fab00901cfbdcfa9fe852d615092"}
Nov 03 21:22:04.050299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.050209    4274 scope.go:122] "RemoveContainer" containerID="45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"
Nov 03 21:22:04.058759 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.058743    4274 scope.go:122] "RemoveContainer" containerID="45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"
Nov 03 21:22:04.059009 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:22:04.058990    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17\": container with ID starting with 45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17 not found: ID does not exist" containerID="45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"
Nov 03 21:22:04.059080 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.059017    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17"} err="failed to get container status \"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17\": rpc error: code = NotFound desc = could not find container \"45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17\": container with ID starting with 45bc420d8f5e2597e8c268c89b9360cef34cca89dc531c9c41d30fa4685a8c17 not found: ID does not exist"
Nov 03 21:22:04.202773 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.202743    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 21:22:04.248788 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:04.248758    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-504eb0-66fcb7f5b7-pb8tf"]
Nov 03 21:22:05.211915 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:05.211881    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" path="/var/lib/kubelet/pods/b891c86c-18e7-4a20-9d8f-af05861c1599/volumes"
Nov 03 21:22:08.304223 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.304197    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 21:22:08.440985 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.440916    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.440985 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.440955    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.440985 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.440974    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441058    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441097    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441130    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441183    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441465 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441269    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441465 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441294    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\") pod \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\" (UID: \"fd0b00b5-fa79-47fd-aee5-710fcb1304ff\") "
Nov 03 21:22:08.441593 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441288    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:08.441654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441422    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:08.441654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441637    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:08.441766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441638    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:08.441874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.441853    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:08.443242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.443211    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:08.443373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.443274    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:08.443373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.443294    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:08.443451 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.443407    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s" pod "fd0b00b5-fa79-47fd-aee5-710fcb1304ff" (UID: "fd0b00b5-fa79-47fd-aee5-710fcb1304ff"). InnerVolumeSpecName "kube-api-access-5sc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:08.542136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542109    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542133    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542145    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sc6s\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-kube-api-access-5sc6s\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542156    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542164    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542173    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542181    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542189    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:08.542322 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:08.542197    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0b00b5-fa79-47fd-aee5-710fcb1304ff-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:09.074535 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.074495    4274 generic.go:356] "Generic (PLEG): container finished" podID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerID="2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016" exitCode=0
Nov 03 21:22:09.074717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.074562    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"
Nov 03 21:22:09.074717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.074568    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerDied","Data":"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"}
Nov 03 21:22:09.074717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.074622    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n" event={"ID":"fd0b00b5-fa79-47fd-aee5-710fcb1304ff","Type":"ContainerDied","Data":"2a65bd0ddd0a21e07a51600dd9ee75222ca3cc24df8ff41d3caaa74a1dce307c"}
Nov 03 21:22:09.074717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.074652    4274 scope.go:122] "RemoveContainer" containerID="2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"
Nov 03 21:22:09.083714 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.083694    4274 scope.go:122] "RemoveContainer" containerID="a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"
Nov 03 21:22:09.093087 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.093063    4274 scope.go:122] "RemoveContainer" containerID="2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"
Nov 03 21:22:09.093387 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:22:09.093365    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016\": container with ID starting with 2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016 not found: ID does not exist" containerID="2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"
Nov 03 21:22:09.093481 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.093400    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016"} err="failed to get container status \"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016\": rpc error: code = NotFound desc = could not find container \"2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016\": container with ID starting with 2e97b1654ca274179ce2d9af03d8705595b920909c214fc7849a24c2fa5c1016 not found: ID does not exist"
Nov 03 21:22:09.093481 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.093424    4274 scope.go:122] "RemoveContainer" containerID="a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"
Nov 03 21:22:09.093691 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:22:09.093674    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce\": container with ID starting with a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce not found: ID does not exist" containerID="a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"
Nov 03 21:22:09.093747 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.093699    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce"} err="failed to get container status \"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce\": rpc error: code = NotFound desc = could not find container \"a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce\": container with ID starting with a02f6c1e7de3110700cca7895e01717efbd761adc3a6f52964957e6c8de612ce not found: ID does not exist"
Nov 03 21:22:09.264000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.263967    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 21:22:09.309997 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:09.309963    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-59t4n"]
Nov 03 21:22:11.214079 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:11.214026    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" path="/var/lib/kubelet/pods/fd0b00b5-fa79-47fd-aee5-710fcb1304ff/volumes"
Nov 03 21:22:13.954811 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:13.954774    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 21:22:13.955274 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:13.954979    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy" containerID="cri-o://7d265ab0ea97960bbe7e55a22db6ea553a7d95cf014c279b7d7d4aabd45f9b00" gracePeriod=30
Nov 03 21:22:19.122413 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.122374    4274 generic.go:356] "Generic (PLEG): container finished" podID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerID="7d265ab0ea97960bbe7e55a22db6ea553a7d95cf014c279b7d7d4aabd45f9b00" exitCode=0
Nov 03 21:22:19.122797 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.122435    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" event={"ID":"e4f122ef-09d2-43d8-a65a-d1389a60964e","Type":"ContainerDied","Data":"7d265ab0ea97960bbe7e55a22db6ea553a7d95cf014c279b7d7d4aabd45f9b00"}
Nov 03 21:22:19.203310 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.203287    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 21:22:19.342405 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342361    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342665 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342636    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:19.342765 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342652    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342765 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342756    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342810    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342889 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342872    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342909    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.342993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.342958    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.343121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.343029    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.343121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.343086    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\") pod \"e4f122ef-09d2-43d8-a65a-d1389a60964e\" (UID: \"e4f122ef-09d2-43d8-a65a-d1389a60964e\") "
Nov 03 21:22:19.343603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.343576    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.343859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.343829    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:19.344269 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.344248    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:19.344606 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.344583    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:19.345816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.345787    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:19.346160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.346015    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:19.346160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.346123    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "kube-api-access-qnglz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:19.348086 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.347127    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:19.351381 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.351357    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token" pod "e4f122ef-09d2-43d8-a65a-d1389a60964e" (UID: "e4f122ef-09d2-43d8-a65a-d1389a60964e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:19.444010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.443983    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444010 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444008    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444019    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444027    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f122ef-09d2-43d8-a65a-d1389a60964e-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444047    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444056    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444064    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:19.444185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:19.444072    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnglz\" (UniqueName: \"kubernetes.io/projected/e4f122ef-09d2-43d8-a65a-d1389a60964e-kube-api-access-qnglz\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:20.128655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:20.128623    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"
Nov 03 21:22:20.129185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:20.128625    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865" event={"ID":"e4f122ef-09d2-43d8-a65a-d1389a60964e","Type":"ContainerDied","Data":"c9e0ff29d4955dae7f056276c844878e9ae55180cd1248434e17c40af4264ca2"}
Nov 03 21:22:20.129185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:20.128803    4274 scope.go:122] "RemoveContainer" containerID="7d265ab0ea97960bbe7e55a22db6ea553a7d95cf014c279b7d7d4aabd45f9b00"
Nov 03 21:22:20.326992 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:20.326928    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 21:22:20.373081 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:20.373051    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-8z865"]
Nov 03 21:22:21.211758 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:21.211719    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" path="/var/lib/kubelet/pods/e4f122ef-09d2-43d8-a65a-d1389a60964e/volumes"
Nov 03 21:22:27.364531 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:22:27.364490    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:22:29.109935 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.109895    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"]
Nov 03 21:22:29.110510 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110490    4274 cpu_manager.go:414] "containerMap: removing container" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110514    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110531    4274 cpu_manager.go:414] "containerMap: removing container" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110541    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110552    4274 cpu_manager.go:414] "containerMap: removing container" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110565    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110572    4274 cpu_manager.go:414] "containerMap: removing container" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110577    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110583    4274 cpu_manager.go:414] "containerMap: removing container" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110588    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.110595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110597    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110601    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110609    4274 cpu_manager.go:414] "containerMap: removing container" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110614    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110620    4274 cpu_manager.go:414] "containerMap: removing container" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110624    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110635    4274 cpu_manager.go:414] "containerMap: removing container" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110639    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110649    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110653    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110663    4274 cpu_manager.go:414] "containerMap: removing container" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110668    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110674    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110678    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110684    4274 cpu_manager.go:414] "containerMap: removing container" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110688    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110696    4274 cpu_manager.go:414] "containerMap: removing container" podUID="31955084-06da-41d1-9adb-5a44bc622645" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110700    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31955084-06da-41d1-9adb-5a44bc622645" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110707    4274 cpu_manager.go:414] "containerMap: removing container" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110711    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110716    4274 cpu_manager.go:414] "containerMap: removing container" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110720    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110727    4274 cpu_manager.go:414] "containerMap: removing container" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110731    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110740    4274 cpu_manager.go:414] "containerMap: removing container" podUID="b3559366-66db-42c9-87a2-27a543ad6816" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110745    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3559366-66db-42c9-87a2-27a543ad6816" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110753    4274 cpu_manager.go:414] "containerMap: removing container" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110757    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110764    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d915e890-e4be-485e-b610-5dba66745dad" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110768    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d915e890-e4be-485e-b610-5dba66745dad" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110775    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110779    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110787    4274 cpu_manager.go:414] "containerMap: removing container" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110791    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110798    4274 cpu_manager.go:414] "containerMap: removing container" podUID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110802    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110812    4274 cpu_manager.go:414] "containerMap: removing container" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110817    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110823    4274 cpu_manager.go:414] "containerMap: removing container" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110827    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110834    4274 cpu_manager.go:414] "containerMap: removing container" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110838    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110848    4274 cpu_manager.go:414] "containerMap: removing container" podUID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110852    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" containerName="main"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110906    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cc0ffc-1bf2-4901-81e6-9a80273f07c5" containerName="maas-api"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110915    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b891c86c-18e7-4a20-9d8f-af05861c1599" containerName="payload-pre-processing"
Nov 03 21:22:29.110901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110922    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c871ba-1376-444f-9242-0e719398a2c0" containerName="payload-processing"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110928    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="47623b5e-6801-4f45-9c81-6715852ab597" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110933    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110938    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110943    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca408a92-57e6-4ec1-9faa-b1d735f4d88d" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110948    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2689076e-c8a2-49fb-8302-6576a600ba29" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110953    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f122ef-09d2-43d8-a65a-d1389a60964e" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110958    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110963    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="de804723-9f46-4882-b35d-e9e36a73880f" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110967    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcae569-67ea-4a3b-95e6-085806de6f58" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110972    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4aeba99-5b4b-49f0-95e1-bf2d2fd60c8f" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110977    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110982    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110987    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ad8f9d-06ff-4fda-8b9f-506634484206" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110994    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0858a92-77a9-4d39-8836-a730a59bc83d" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.110998    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fadca4-3d39-460b-9ee4-01aff0c65391" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111002    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1561b796-6182-4059-a706-9b53b22506b0" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111007    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="31955084-06da-41d1-9adb-5a44bc622645" containerName="maas-api"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111011    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b80b7f-fbf8-44b4-9be5-035cb061d4e6" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111015    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="248f67e5-66b7-4b53-9e09-86a28e073482" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111019    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3559366-66db-42c9-87a2-27a543ad6816" containerName="maas-api"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111024    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111028    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74b724f-1f77-41d3-beef-430899abeb4d" containerName="istio-proxy"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111050    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d915e890-e4be-485e-b610-5dba66745dad" containerName="maas-api"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111058    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3627eb01-f4b2-4012-8cc9-a6f0383bc610" containerName="main"
Nov 03 21:22:29.112273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.111063    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a39c18-5bcf-4ddf-b436-9b507c712d74" containerName="payload-pre-processing"
Nov 03 21:22:29.115403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.115387    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224631    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224664 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224666    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224907 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224689    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224907 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224823    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224920    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.224998 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.224966    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwll9\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-kube-api-access-bwll9\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.225107 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.225024    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.225107 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.225066    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.225184 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.225110    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.306949 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.306917    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"]
Nov 03 21:22:29.325565 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325535    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325589    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325615    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325644    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325678    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325740    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.325809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325786    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwll9\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-kube-api-access-bwll9\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326147 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325818    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326147 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.325848    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326341 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.326320    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.326381    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326622 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.326604    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.326698 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.326667    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.327195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.327176    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.328477 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.328443    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.328766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.328749    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.383367 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.383312    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.383456 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.383409    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwll9\" (UniqueName: \"kubernetes.io/projected/39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6-kube-api-access-bwll9\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf\" (UID: \"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.427426 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.427407    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:29.838430 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.838405    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"]
Nov 03 21:22:29.840533 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:22:29.840505    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ffac5b_5fae_4192_8bc3_e6fa7e84d1c6.slice/crio-2ec49358562415bfc3872e2b598bc3b0bc7e81bec960647152f38a0764987e5b WatchSource:0}: Error finding container 2ec49358562415bfc3872e2b598bc3b0bc7e81bec960647152f38a0764987e5b: Status 404 returned error can't find the container with id 2ec49358562415bfc3872e2b598bc3b0bc7e81bec960647152f38a0764987e5b
Nov 03 21:22:29.842620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.842590    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:29.842691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.842669    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:29.842729 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:29.842697    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:30.173006 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.172915    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" event={"ID":"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6","Type":"ContainerStarted","Data":"49367b87faf425309cc30b65eb7c9588c09ea21343d70fc97e08965153dd9733"}
Nov 03 21:22:30.173006 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.172983    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" event={"ID":"39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6","Type":"ContainerStarted","Data":"2ec49358562415bfc3872e2b598bc3b0bc7e81bec960647152f38a0764987e5b"}
Nov 03 21:22:30.287613 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.287549    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podStartSLOduration=1.28753549 podStartE2EDuration="1.28753549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:30.285395353 +0000 UTC m=+3175.211965591" watchObservedRunningTime="2026-11-03 21:22:30.28753549 +0000 UTC m=+3175.214105728"
Nov 03 21:22:30.428632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.428541    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:30.430138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.430114    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:30.430250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:30.430160    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:31.427776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:31.427737    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:31.428219 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:31.427790    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:32.428349 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:32.428311    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:32.428819 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:32.428368    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:33.428066 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:33.428019    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:33.428240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:33.428087    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:34.428663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:34.428632    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:34.429068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:34.428701    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:35.428555 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:35.428516    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:35.428726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:35.428577    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:36.427976 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:36.427901    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 21:22:36.427976 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:36.427954    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 21:22:38.428753 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:38.428707    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:22:38.429293 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:38.428786    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:22:39.470217 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:39.470170    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:22:39.470734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:39.470246    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:22:40.186030 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:40.186003    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:40.186340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:40.186318    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:40.186997 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:40.186977    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf"
Nov 03 21:22:44.220467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220437    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"]
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220765    4274 cpu_manager.go:414] "containerMap: removing container" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220776    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220804    4274 cpu_manager.go:414] "containerMap: removing container" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220809    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0b00b5-fa79-47fd-aee5-710fcb1304ff" containerName="istio-proxy"
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220822    4274 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:44.220859 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220827    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:44.221105 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220877    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0a315-aaf8-4a5e-abf5-345fd81b5bea" containerName="istio-proxy"
Nov 03 21:22:44.221105 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.220888    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f781bc-3dfa-45bf-b7ce-8d19954a8aa0" containerName="cleanup"
Nov 03 21:22:44.225207 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.225187    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278453 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278422    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278585 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278455    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp955\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-kube-api-access-jp955\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278585 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278481    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278625    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278692 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278656    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278720    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278746    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278824 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278811    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.278864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.278856    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc94016b-79d8-4dd3-a968-75b7ad182092-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.322305 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.322275    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"]
Nov 03 21:22:44.379875 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.379839    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.379878    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.379931    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.379992    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc94016b-79d8-4dd3-a968-75b7ad182092-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380026    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380064    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp955\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-kube-api-access-jp955\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380106    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380161    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380183    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380547 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380488    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380638 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380576    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380709    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380845    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc94016b-79d8-4dd3-a968-75b7ad182092-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.380960 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.380939    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.382495 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.382469    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.382589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.382574    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.439597 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.439570    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp955\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-kube-api-access-jp955\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.439684 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.439647    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc94016b-79d8-4dd3-a968-75b7ad182092-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4\" (UID: \"cc94016b-79d8-4dd3-a968-75b7ad182092\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.537138 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.537101    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:44.947947 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.947908    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"]
Nov 03 21:22:44.949685 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:22:44.949654    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc94016b_79d8_4dd3_a968_75b7ad182092.slice/crio-ea7c1e48c8509d7fc5b653338279e6d8b9ba96db2392c9387c6298b15c2e27b9 WatchSource:0}: Error finding container ea7c1e48c8509d7fc5b653338279e6d8b9ba96db2392c9387c6298b15c2e27b9: Status 404 returned error can't find the container with id ea7c1e48c8509d7fc5b653338279e6d8b9ba96db2392c9387c6298b15c2e27b9
Nov 03 21:22:44.952226 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.952195    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:44.952303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.952259    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:44.952303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:44.952288    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:45.251263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:45.251187    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" event={"ID":"cc94016b-79d8-4dd3-a968-75b7ad182092","Type":"ContainerStarted","Data":"a8356c738b68edb8b59f56cdb11fff1ba75b36267f300ed6e726e1b8157ad28d"}
Nov 03 21:22:45.251263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:45.251238    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" event={"ID":"cc94016b-79d8-4dd3-a968-75b7ad182092","Type":"ContainerStarted","Data":"ea7c1e48c8509d7fc5b653338279e6d8b9ba96db2392c9387c6298b15c2e27b9"}
Nov 03 21:22:45.364314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:45.364251    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podStartSLOduration=1.364234659 podStartE2EDuration="1.364234659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:45.363754769 +0000 UTC m=+3190.290325007" watchObservedRunningTime="2026-11-03 21:22:45.364234659 +0000 UTC m=+3190.290804899"
Nov 03 21:22:45.537587 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:45.537555    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:46.539484 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:46.539451    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:46.539962 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:46.539497    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:47.538244 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:47.538209    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:47.538472 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:47.538274    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:48.537844 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:48.537812    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:48.538350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:48.537861    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:49.538052 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:49.537999    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:49.538537 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:49.538092    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:50.537641 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:50.537602    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:50.537919 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:50.537664    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:51.537526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:51.537497    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 21:22:51.537922 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:51.537543    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 21:22:53.169677 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:53.169635    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:53.170146 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:53.169957    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:53.170868 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:22:53.170844    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4"
Nov 03 21:22:57.364468 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:22:57.364427    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:23:27.365319 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:23:27.365281    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:23:44.696535 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.696502    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"]
Nov 03 21:23:44.699863 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.699847    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826023 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.825988    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826023 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826024    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826225 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826097    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrzx\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-kube-api-access-8qrzx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826225 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826181    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826225 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826210    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826334 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826227    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc1dc11-162e-45ca-9cce-2984487898d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826382 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826361    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826420 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826396    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.826420 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.826414    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.896209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.896180    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"]
Nov 03 21:23:44.927027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.926997    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927031    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc1dc11-162e-45ca-9cce-2984487898d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927116    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927147    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927168    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927255 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927240    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927270    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927317    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrzx\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-kube-api-access-8qrzx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927365    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927458    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927680 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927472    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927758 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927736    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.927826 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.927775    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.928136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.928110    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc1dc11-162e-45ca-9cce-2984487898d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.929776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.929756    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.929861 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.929833    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.983832 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.983777    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:44.983909 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:44.983861    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrzx\" (UniqueName: \"kubernetes.io/projected/dcc1dc11-162e-45ca-9cce-2984487898d8-kube-api-access-8qrzx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xclk7\" (UID: \"dcc1dc11-162e-45ca-9cce-2984487898d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:45.011605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.011586    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:45.426025 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.425997    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"]
Nov 03 21:23:45.427950 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:23:45.427910    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc1dc11_162e_45ca_9cce_2984487898d8.slice/crio-b8eac0330373ab2bb64b3d666c761e749c94b3c5df5fb954c8ef5601a9eae7b5 WatchSource:0}: Error finding container b8eac0330373ab2bb64b3d666c761e749c94b3c5df5fb954c8ef5601a9eae7b5: Status 404 returned error can't find the container with id b8eac0330373ab2bb64b3d666c761e749c94b3c5df5fb954c8ef5601a9eae7b5
Nov 03 21:23:45.430213 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.430177    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:45.430304 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.430260    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:45.430351 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.430304    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:45.541411 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.541376    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" event={"ID":"dcc1dc11-162e-45ca-9cce-2984487898d8","Type":"ContainerStarted","Data":"7c8406ac309464679690e21a6eab3dba848d8b0f8083a7f2266d72980de41ed4"}
Nov 03 21:23:45.541586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.541432    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" event={"ID":"dcc1dc11-162e-45ca-9cce-2984487898d8","Type":"ContainerStarted","Data":"b8eac0330373ab2bb64b3d666c761e749c94b3c5df5fb954c8ef5601a9eae7b5"}
Nov 03 21:23:45.654060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:45.653995    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podStartSLOduration=1.653982759 podStartE2EDuration="1.653982759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:45.653426182 +0000 UTC m=+3250.579996419" watchObservedRunningTime="2026-11-03 21:23:45.653982759 +0000 UTC m=+3250.580552996"
Nov 03 21:23:46.012472 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:46.012439    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:46.013900 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:46.013875    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:46.014020 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:46.013923    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:47.012509 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:47.012472    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:47.012903 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:47.012536    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:48.012472 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:48.012439    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:48.012646 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:48.012495    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:49.012478 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:49.012439    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:49.012648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:49.012510    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:50.012679 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:50.012643    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:50.013100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:50.012698    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:51.012168 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:51.012128    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:51.012355 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:51.012198    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:52.012439 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:52.012402    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 21:23:52.012912 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:52.012459    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 21:23:53.567670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:53.567637    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:53.568195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:53.568056    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:53.568731 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:23:53.568710    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7"
Nov 03 21:23:57.366291 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:23:57.366254    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:24:27.365539 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:24:27.365501    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:24:57.364749 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:24:57.364695    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:25:27.368750 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:25:27.368713    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:25:39.218632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:25:39.218216    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:25:39.218632 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:25:39.218280    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4" podUID="cc94016b-79d8-4dd3-a968-75b7ad182092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:25:39.629068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:25:39.627749    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:25:39.629068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:25:39.627809    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xclk7" podUID="dcc1dc11-162e-45ca-9cce-2984487898d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:25:57.364995 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:25:57.364955    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:26:27.364544 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:26:27.364504    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:26:41.260058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:26:41.248399    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:26:41.260058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:26:41.248491    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:26:57.364898 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:26:57.364861    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:27:27.365276 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:27:27.365239    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:27:57.364611 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:27:57.364572    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:28:27.364974 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:28:27.364940    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:28:57.364929 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:28:57.364896    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:29:27.365509 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:29:27.365473    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:29:57.365409 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:29:57.365372    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:30:27.364667 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:30:27.364630    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:30:31.556403 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.556367    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:30:31.560241 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.560204    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.609184 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.609160    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-357e69-openshift-default-dockercfg-mkkhs\""
Nov 03 21:30:31.637891 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.637858    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.637927    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638003 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.637989    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638018    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638117 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638049    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638232 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638216    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638274 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638251    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8hj\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638365 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638352    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.638406 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.638374    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.661397 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.661360    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:30:31.739406 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739375    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739413    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8hj\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739459    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739486    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739589 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739563    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739808 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739613    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739808 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739643    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739808 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739666    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.739808 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739688    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.740165 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.739929    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.740165 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.740109    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.740346 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.740242    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.740404 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.740356    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.740568 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.740546    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.742020 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.741999    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.742277 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.742257    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.796809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.796772    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.796930 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.796837    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8hj\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\") pod \"e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") " pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:31.873007 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:31.872926    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:32.290294 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.290267    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:30:32.292524 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:30:32.292491    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdeb6ec_3d22_4c46_82a6_2131ac98a579.slice/crio-a4a29a23103f70bf541801a65a0d4485a51184bba35f5dee0d785ed65647d64c WatchSource:0}: Error finding container a4a29a23103f70bf541801a65a0d4485a51184bba35f5dee0d785ed65647d64c: Status 404 returned error can't find the container with id a4a29a23103f70bf541801a65a0d4485a51184bba35f5dee0d785ed65647d64c
Nov 03 21:30:32.294948 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.294917    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:32.295014 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.294982    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:32.295068 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.295011    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:32.649336 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.649234    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerStarted","Data":"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"}
Nov 03 21:30:32.649336 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.649289    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerStarted","Data":"a4a29a23103f70bf541801a65a0d4485a51184bba35f5dee0d785ed65647d64c"}
Nov 03 21:30:32.762701 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.762645    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podStartSLOduration=1.762628225 podStartE2EDuration="1.762628225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:32.761627974 +0000 UTC m=+3657.688198211" watchObservedRunningTime="2026-11-03 21:30:32.762628225 +0000 UTC m=+3657.689198469"
Nov 03 21:30:32.873569 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:32.873530    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:33.879021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:33.878991    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:34.657869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:34.657839    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:34.658649 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:34.658629    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:30:39.721741 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.721703    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:30:39.725244 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.725221    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.774397 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.774365    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-357e69\""
Nov 03 21:30:39.819715 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.819683    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546fw\" (UniqueName: \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.819855 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.819790    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.820288 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.820269    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5056388c-dockercfg-dswqw\""
Nov 03 21:30:39.822149 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.822127    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:30:39.920763 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.920735    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546fw\" (UniqueName: \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.920868 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.920843    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.921531 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.921509    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:39.978276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:39.978225    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546fw\" (UniqueName: \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\") pod \"payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:40.038428 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.038393    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:40.402005 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.401980    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:30:40.403879 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:30:40.403849    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e52de38_5f94_42f2_aeb6_53b5cc0ea735.slice/crio-6a2b0045309676217b78393dc78ac7daee8394d239f2af634fcc517940423018 WatchSource:0}: Error finding container 6a2b0045309676217b78393dc78ac7daee8394d239f2af634fcc517940423018: Status 404 returned error can't find the container with id 6a2b0045309676217b78393dc78ac7daee8394d239f2af634fcc517940423018
Nov 03 21:30:40.684461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.684379    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" event={"ID":"7e52de38-5f94-42f2-aeb6-53b5cc0ea735","Type":"ContainerStarted","Data":"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"}
Nov 03 21:30:40.684461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.684433    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" event={"ID":"7e52de38-5f94-42f2-aeb6-53b5cc0ea735","Type":"ContainerStarted","Data":"6a2b0045309676217b78393dc78ac7daee8394d239f2af634fcc517940423018"}
Nov 03 21:30:40.684461 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.684461    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:40.793270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:40.793216    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" podStartSLOduration=1.7932035530000001 podStartE2EDuration="1.793203553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:40.792959953 +0000 UTC m=+3665.719530189" watchObservedRunningTime="2026-11-03 21:30:40.793203553 +0000 UTC m=+3665.719773791"
Nov 03 21:30:51.693270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:30:51.693240    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:30:57.364704 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:30:57.364616    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:31:00.795885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.795852    4274 generic.go:356] "Generic (PLEG): container finished" podID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerID="0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a" exitCode=0
Nov 03 21:31:00.796329 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.795925    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerDied","Data":"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"}
Nov 03 21:31:00.796329 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.796319    4274 scope.go:122] "RemoveContainer" containerID="0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"
Nov 03 21:31:00.797424 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.797399    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:00.797493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.797478    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:00.797530 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:00.797516    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:01.802261 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:01.802226    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerStarted","Data":"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"}
Nov 03 21:31:01.873422 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:01.873391    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:31:01.874614 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:01.874590    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:01.874681 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:01.874663    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:02.874060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:02.873496    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:02.874060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:02.873561    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:03.701717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.701682    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:31:03.705333 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.705310    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.791648 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.791612    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwb6\" (UniqueName: \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.791822 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.791725    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.791822 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.791790    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.791935 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.791821    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.792008 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.791970    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.792089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.792073    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.801631 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.801603    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-357e69\"/\"openshift-service-ca.crt\""
Nov 03 21:31:03.801631 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.801603    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-357e69\"/\"simulator-single-main-357e69-kserve-self-signed-certs\""
Nov 03 21:31:03.801806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.801614    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-357e69\"/\"kube-root-ca.crt\""
Nov 03 21:31:03.801806 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.801741    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-357e69\"/\"default-dockercfg-djfs6\""
Nov 03 21:31:03.803762 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.803742    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:31:03.879391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.879352    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:03.879872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.879410    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:03.892624 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892591    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.892766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892642    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.892766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892677    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.892766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892734    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.892766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892763    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwb6\" (UniqueName: \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.892976 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.892942    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.893179 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.893156    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.893263 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.893174    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.893355 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.893338    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.895023 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.894999    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.895231 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.895210    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:03.970788 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:03.970701    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwb6\" (UniqueName: \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\") pod \"simulator-single-main-357e69-kserve-8557684f87-lk256\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") " pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:04.016707 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.016678    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:04.402371 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.402344    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:31:04.404148 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:31:04.404116    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod348176dd_a267_4122_b3f0_5c22c2be7971.slice/crio-383f96204d6dba6a935572ff50e6e2ac93a4f1b1d7bffad6b4917ed4cbd9a643 WatchSource:0}: Error finding container 383f96204d6dba6a935572ff50e6e2ac93a4f1b1d7bffad6b4917ed4cbd9a643: Status 404 returned error can't find the container with id 383f96204d6dba6a935572ff50e6e2ac93a4f1b1d7bffad6b4917ed4cbd9a643
Nov 03 21:31:04.666116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.660129    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:31:04.816977 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.816938    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" event={"ID":"348176dd-a267-4122-b3f0-5c22c2be7971","Type":"ContainerStarted","Data":"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"}
Nov 03 21:31:04.816977 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.816979    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" event={"ID":"348176dd-a267-4122-b3f0-5c22c2be7971","Type":"ContainerStarted","Data":"383f96204d6dba6a935572ff50e6e2ac93a4f1b1d7bffad6b4917ed4cbd9a643"}
Nov 03 21:31:04.873707 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.873662    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:04.873893 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.873714    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:04.931343 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:04.931249    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" podStartSLOduration=1.931236572 podStartE2EDuration="1.931236572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:04.930195547 +0000 UTC m=+3689.856765797" watchObservedRunningTime="2026-11-03 21:31:04.931236572 +0000 UTC m=+3689.857806818"
Nov 03 21:31:05.017726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:05.017694    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:05.873711 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:05.873673    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:05.873879 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:05.873728    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:06.873783 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:06.873743    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 03 21:31:06.874213 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:06.873815    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 03 21:31:07.877584 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:07.877555    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:31:07.878563 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:07.878542    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:31:14.017709 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:14.017672    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:14.030347 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:14.030323    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:14.873705 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:31:14.873669    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:31:27.364393 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:31:27.364355    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:31:57.364829 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:31:57.364793    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:32:05.026230 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:05.026192    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:32:12.362412 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:12.362378    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:32:17.362904 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:17.362865    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing" containerID="cri-o://a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e" gracePeriod=25
Nov 03 21:32:20.046000 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.045957    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" podUID="348176dd-a267-4122-b3f0-5c22c2be7971" containerName="main" containerID="cri-o://8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1" gracePeriod=45
Nov 03 21:32:20.292290 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.292268    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:32:20.396775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.396699    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\" (UniqueName: \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.396775 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.396736    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\" (UniqueName: \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.396954 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.396825    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.396954 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.396903    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.397054 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.396968    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.397054 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397000    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\") pod \"348176dd-a267-4122-b3f0-5c22c2be7971\" (UID: \"348176dd-a267-4122-b3f0-5c22c2be7971\") "
Nov 03 21:32:20.397231 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397180    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:20.397353 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397259    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:20.397353 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397272    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:20.397467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397442    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:20.397523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397468    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:20.397523 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.397483    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:20.399072 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.399046    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "kube-api-access-kvwb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:20.399178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.399105    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:32:20.399178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.399113    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm" pod "348176dd-a267-4122-b3f0-5c22c2be7971" (UID: "348176dd-a267-4122-b3f0-5c22c2be7971"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:20.498342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.498308    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348176dd-a267-4122-b3f0-5c22c2be7971-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:20.498342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.498338    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvwb6\" (UniqueName: \"kubernetes.io/projected/348176dd-a267-4122-b3f0-5c22c2be7971-kube-api-access-kvwb6\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:20.498342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:20.498350    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348176dd-a267-4122-b3f0-5c22c2be7971-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:21.156769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.156734    4274 generic.go:356] "Generic (PLEG): container finished" podID="348176dd-a267-4122-b3f0-5c22c2be7971" containerID="8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1" exitCode=0
Nov 03 21:32:21.157267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.156782    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"
Nov 03 21:32:21.157267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.156782    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" event={"ID":"348176dd-a267-4122-b3f0-5c22c2be7971","Type":"ContainerDied","Data":"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"}
Nov 03 21:32:21.157267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.156897    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256" event={"ID":"348176dd-a267-4122-b3f0-5c22c2be7971","Type":"ContainerDied","Data":"383f96204d6dba6a935572ff50e6e2ac93a4f1b1d7bffad6b4917ed4cbd9a643"}
Nov 03 21:32:21.157267 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.156922    4274 scope.go:122] "RemoveContainer" containerID="8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"
Nov 03 21:32:21.166221 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.166180    4274 scope.go:122] "RemoveContainer" containerID="8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"
Nov 03 21:32:21.166477 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:21.166458    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1\": container with ID starting with 8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1 not found: ID does not exist" containerID="8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"
Nov 03 21:32:21.166535 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.166485    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1"} err="failed to get container status \"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1\": rpc error: code = NotFound desc = could not find container \"8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1\": container with ID starting with 8b4f39a742a9bf87fc85fa8f3ba852ce585e0ba1e912c781688cf995694025e1 not found: ID does not exist"
Nov 03 21:32:21.309884 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.309856    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:32:21.355797 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.355769    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-357e69/simulator-single-main-357e69-kserve-8557684f87-lk256"]
Nov 03 21:32:21.690647 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:21.690617    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:32:23.215792 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:23.215751    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348176dd-a267-4122-b3f0-5c22c2be7971" path="/var/lib/kubelet/pods/348176dd-a267-4122-b3f0-5c22c2be7971/volumes"
Nov 03 21:32:24.201044 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:24.201002    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:32:24.201297 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:24.201214    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy" containerID="cri-o://563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633" gracePeriod=30
Nov 03 21:32:27.365166 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:27.365130    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:32:29.454302 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.454279    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:32:29.474308 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474246    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474308 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474294    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474320    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474344    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474364    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474402    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474470    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474538    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474578    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\") pod \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\" (UID: \"8bdeb6ec-3d22-4c46-82a6-2131ac98a579\") "
Nov 03 21:32:29.474751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474596    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:29.474751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474661    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:29.474751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474678    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:29.475012 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474829    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:29.475012 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.474994    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.475012 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.475005    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:29.475160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.475017    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.475160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.475031    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.475160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.475064    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.476741 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.476715    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:29.476837 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.476757    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "kube-api-access-wt8hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:29.476890 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.476856    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:32:29.476890 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.476865    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy" pod "8bdeb6ec-3d22-4c46-82a6-2131ac98a579" (UID: "8bdeb6ec-3d22-4c46-82a6-2131ac98a579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:32:29.578178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.578147    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8hj\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-kube-api-access-wt8hj\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.578178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.578178    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.578369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.578188    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.578369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.578197    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:29.578369 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:29.578210    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdeb6ec-3d22-4c46-82a6-2131ac98a579-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:30.198894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.198857    4274 generic.go:356] "Generic (PLEG): container finished" podID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerID="563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633" exitCode=0
Nov 03 21:32:30.199116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.198923    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"
Nov 03 21:32:30.199116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.198945    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerDied","Data":"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"}
Nov 03 21:32:30.199116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.199026    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm" event={"ID":"8bdeb6ec-3d22-4c46-82a6-2131ac98a579","Type":"ContainerDied","Data":"a4a29a23103f70bf541801a65a0d4485a51184bba35f5dee0d785ed65647d64c"}
Nov 03 21:32:30.199116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.199073    4274 scope.go:122] "RemoveContainer" containerID="563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"
Nov 03 21:32:30.208609 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.208593    4274 scope.go:122] "RemoveContainer" containerID="0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"
Nov 03 21:32:30.216583 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.216568    4274 scope.go:122] "RemoveContainer" containerID="563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"
Nov 03 21:32:30.216835 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:30.216813    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633\": container with ID starting with 563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633 not found: ID does not exist" containerID="563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"
Nov 03 21:32:30.216885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.216843    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633"} err="failed to get container status \"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633\": rpc error: code = NotFound desc = could not find container \"563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633\": container with ID starting with 563fc3376e8063a8e6210f243ec902f98ffdd535e82010a4507754ad6a1f6633 not found: ID does not exist"
Nov 03 21:32:30.216885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.216858    4274 scope.go:122] "RemoveContainer" containerID="0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"
Nov 03 21:32:30.217104 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:30.217085    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a\": container with ID starting with 0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a not found: ID does not exist" containerID="0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"
Nov 03 21:32:30.217158 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.217113    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a"} err="failed to get container status \"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a\": rpc error: code = NotFound desc = could not find container \"0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a\": container with ID starting with 0380e7862fc31bf44fd76d0a3e0e4e35a72757f840c3ea479a68bf9bb3e02b7a not found: ID does not exist"
Nov 03 21:32:30.400679 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.400645    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:32:30.404843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:30.404815    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-357e69-openshift-default-678bb66ffchqqkm"]
Nov 03 21:32:31.211882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:31.211843    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" path="/var/lib/kubelet/pods/8bdeb6ec-3d22-4c46-82a6-2131ac98a579/volumes"
Nov 03 21:32:31.690966 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:31.690930    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:32:37.611265 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.611239    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:32:37.652696 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.652638    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\" (UniqueName: \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\") pod \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") "
Nov 03 21:32:37.652696 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.652672    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\") pod \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\" (UID: \"7e52de38-5f94-42f2-aeb6-53b5cc0ea735\") "
Nov 03 21:32:37.653089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.653066    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume" pod "7e52de38-5f94-42f2-aeb6-53b5cc0ea735" (UID: "7e52de38-5f94-42f2-aeb6-53b5cc0ea735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:37.654631 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.654603    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw" pod "7e52de38-5f94-42f2-aeb6-53b5cc0ea735" (UID: "7e52de38-5f94-42f2-aeb6-53b5cc0ea735"). InnerVolumeSpecName "kube-api-access-546fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:37.754332 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.754300    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546fw\" (UniqueName: \"kubernetes.io/projected/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-kube-api-access-546fw\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:37.754332 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:37.754327    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e52de38-5f94-42f2-aeb6-53b5cc0ea735-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:38.235828 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.235792    4274 generic.go:356] "Generic (PLEG): container finished" podID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerID="a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e" exitCode=0
Nov 03 21:32:38.236011 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.235837    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"
Nov 03 21:32:38.236011 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.235872    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" event={"ID":"7e52de38-5f94-42f2-aeb6-53b5cc0ea735","Type":"ContainerDied","Data":"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"}
Nov 03 21:32:38.236011 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.235919    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz" event={"ID":"7e52de38-5f94-42f2-aeb6-53b5cc0ea735","Type":"ContainerDied","Data":"6a2b0045309676217b78393dc78ac7daee8394d239f2af634fcc517940423018"}
Nov 03 21:32:38.236011 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.235942    4274 scope.go:122] "RemoveContainer" containerID="a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"
Nov 03 21:32:38.244822 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.244807    4274 scope.go:122] "RemoveContainer" containerID="a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"
Nov 03 21:32:38.245109 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:38.245090    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e\": container with ID starting with a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e not found: ID does not exist" containerID="a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"
Nov 03 21:32:38.245160 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.245119    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e"} err="failed to get container status \"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e\": rpc error: code = NotFound desc = could not find container \"a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e\": container with ID starting with a82132d26fa762eac9f5fc1248d618400c4eb648f043cf9f8acdde2868eb965e not found: ID does not exist"
Nov 03 21:32:38.387903 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.387866    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:32:38.433736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:38.433706    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-357e69-848bdljhvz"]
Nov 03 21:32:39.213223 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:39.213187    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" path="/var/lib/kubelet/pods/7e52de38-5f94-42f2-aeb6-53b5cc0ea735/volumes"
Nov 03 21:32:42.523060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:42.522123    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:42.523060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:32:42.522185    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-sfss5" podUID="be93de48-76dc-499d-905d-7d7585cde6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:57.364315 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:32:57.364276    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:33:27.364860 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:33:27.364827    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:33:57.364819 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:33:57.364777    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:34:26.234185 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:34:26.234137    4274 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:34:26.234852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:34:26.234210    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf" podUID="39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:27.364763 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:34:27.364730    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:34:57.365217 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:34:57.365178    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:35:10.665153 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665119    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:35:10.665670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665650    4274 cpu_manager.go:414] "containerMap: removing container" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665674    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665711    4274 cpu_manager.go:414] "containerMap: removing container" podUID="348176dd-a267-4122-b3f0-5c22c2be7971" containerName="main"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665719    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348176dd-a267-4122-b3f0-5c22c2be7971" containerName="main"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665738    4274 cpu_manager.go:414] "containerMap: removing container" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665746    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.665760 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665761    4274 cpu_manager.go:414] "containerMap: removing container" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.666082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665769    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.666082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665871    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e52de38-5f94-42f2-aeb6-53b5cc0ea735" containerName="payload-pre-processing"
Nov 03 21:35:10.666082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665889    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="348176dd-a267-4122-b3f0-5c22c2be7971" containerName="main"
Nov 03 21:35:10.666082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665902    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.666082 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.665914    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdeb6ec-3d22-4c46-82a6-2131ac98a579" containerName="istio-proxy"
Nov 03 21:35:10.669027 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.669004    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.717907 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.717876    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7b11dd-openshift-default-dockercfg-kr2bn\""
Nov 03 21:35:10.727240 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727214    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727258    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727340 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727279    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727431 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727363    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727473 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727449    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727478    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727556 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727549    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727576    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbg74\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.727660 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.727601    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.768200 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.768173    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:35:10.828421 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828392    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828434    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbg74\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828457    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828579 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828485    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828612    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828661    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828851 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828755    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828851 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828822    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828851 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828838    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828996 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828872    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.828996 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.828932    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.829142 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.829097    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.829352 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.829331    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.829429 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.829418    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.831084 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.831061    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.831404 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.831383    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.886354 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.886315    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.886540 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.886458    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbg74\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\") pod \"e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") " pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:10.981683 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:10.981600    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:11.393682 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:11.393659    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:35:11.395742 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:35:11.395706    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0a2d5c_47f2_4d0f_9bfb_6d8090f31d4a.slice/crio-c03c9fd1a3ef74e6244e834f25299a94ae030ff3410d779d23dea92b3b771eb0 WatchSource:0}: Error finding container c03c9fd1a3ef74e6244e834f25299a94ae030ff3410d779d23dea92b3b771eb0: Status 404 returned error can't find the container with id c03c9fd1a3ef74e6244e834f25299a94ae030ff3410d779d23dea92b3b771eb0
Nov 03 21:35:11.397986 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:11.397955    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:11.398078 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:11.398022    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:11.398132 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:11.398079    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:12.034809 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:12.034767    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerStarted","Data":"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"}
Nov 03 21:35:12.035210 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:12.034871    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerStarted","Data":"c03c9fd1a3ef74e6244e834f25299a94ae030ff3410d779d23dea92b3b771eb0"}
Nov 03 21:35:12.195880 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:12.195829    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podStartSLOduration=2.195816629 podStartE2EDuration="2.195816629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:12.19383243 +0000 UTC m=+3937.120402670" watchObservedRunningTime="2026-11-03 21:35:12.195816629 +0000 UTC m=+3937.122386926"
Nov 03 21:35:12.982768 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:12.982731    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:12.987421 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:12.987395    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:13.039950 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:13.039923    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:13.040658 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:13.040641    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:27.367537 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:35:27.367500    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:35:39.155874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.155782    4274 generic.go:356] "Generic (PLEG): container finished" podID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerID="43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8" exitCode=0
Nov 03 21:35:39.155874 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.155818    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerDied","Data":"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"}
Nov 03 21:35:39.156372 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.156159    4274 scope.go:122] "RemoveContainer" containerID="43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"
Nov 03 21:35:39.157318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.157290    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:39.157387 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.157365    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:39.157428 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:39.157399    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:35:40.175519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:40.175488    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerStarted","Data":"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"}
Nov 03 21:35:40.982178 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:40.982142    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:40.983303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:40.983280    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:35:40.983376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:40.983334    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 21:35:41.982292 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:41.982257    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:35:41.982691 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:41.982310    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 21:35:42.982139 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:42.982097    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:35:42.982299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:42.982158    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 21:35:43.040314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.040280    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:43.161426 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.161391    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:35:43.173765 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.173735    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191409 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191385    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191512 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191431    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191563 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191528    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191563 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191550    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191659 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191581    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.191745 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.191724    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnbf\" (UniqueName: \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.216719 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.216688    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:35:43.267120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.263347    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7b11dd\"/\"default-dockercfg-lhc85\""
Nov 03 21:35:43.267120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.263517    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7b11dd\"/\"kube-root-ca.crt\""
Nov 03 21:35:43.267120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.263589    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7b11dd\"/\"openshift-service-ca.crt\""
Nov 03 21:35:43.270235 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.270197    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7b11dd\"/\"simulator-single-main-7b11dd-kserve-self-signed-certs\""
Nov 03 21:35:43.292652 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.292625    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.292807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.292674    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.292807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.292721    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.292807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.292780    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnbf\" (UniqueName: \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.292987 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.292877    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.293137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.293113    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.293218 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.293150    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.293304 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.293278    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.293574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.293552    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.295197 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.295181    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.295333 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.295316    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.370726 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.370693    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnbf\" (UniqueName: \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\") pod \"simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") " pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.485361 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.485324    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:43.868113 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.867493    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:35:43.874421 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:35:43.874394    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5e335c_3073_43af_a9f8_36358f987977.slice/crio-88b72345e2258241c9ed0bb8e8b2bb93dc74bfd60bfe078eb16ad0948b5fbda6 WatchSource:0}: Error finding container 88b72345e2258241c9ed0bb8e8b2bb93dc74bfd60bfe078eb16ad0948b5fbda6: Status 404 returned error can't find the container with id 88b72345e2258241c9ed0bb8e8b2bb93dc74bfd60bfe078eb16ad0948b5fbda6
Nov 03 21:35:43.982778 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.982745    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:35:43.982943 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:43.982806    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 21:35:44.201303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.201217    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" event={"ID":"ca5e335c-3073-43af-a9f8-36358f987977","Type":"ContainerStarted","Data":"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"}
Nov 03 21:35:44.201303 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.201261    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" event={"ID":"ca5e335c-3073-43af-a9f8-36358f987977","Type":"ContainerStarted","Data":"88b72345e2258241c9ed0bb8e8b2bb93dc74bfd60bfe078eb16ad0948b5fbda6"}
Nov 03 21:35:44.317564 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.317486    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" podStartSLOduration=1.317470103 podStartE2EDuration="1.317470103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:44.31734517 +0000 UTC m=+3969.243915407" watchObservedRunningTime="2026-11-03 21:35:44.317470103 +0000 UTC m=+3969.244040341"
Nov 03 21:35:44.485605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.485533    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:44.982124 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.982088    4274 patch_prober.go:29] interesting pod/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 21:35:44.982300 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:44.982138    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 21:35:45.985940 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:45.985908    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:45.986766 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:45.986745    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:35:53.486497 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:53.486460    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:53.499367 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:53.499340    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:54.257071 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:35:54.257029    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:35:57.364808 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:35:57.364772    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:36:27.366725 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:36:27.366686    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:36:57.365049 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:36:57.364993    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:37:00.290208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:00.290171    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:37:15.310883 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.310824    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" podUID="ca5e335c-3073-43af-a9f8-36358f987977" containerName="main" containerID="cri-o://906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21" gracePeriod=45
Nov 03 21:37:15.546192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.546168    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:37:15.605846 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.605785    4274 generic.go:356] "Generic (PLEG): container finished" podID="ca5e335c-3073-43af-a9f8-36358f987977" containerID="906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21" exitCode=0
Nov 03 21:37:15.605846 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.605820    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"
Nov 03 21:37:15.605846 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.605828    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" event={"ID":"ca5e335c-3073-43af-a9f8-36358f987977","Type":"ContainerDied","Data":"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"}
Nov 03 21:37:15.606085 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.605868    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p" event={"ID":"ca5e335c-3073-43af-a9f8-36358f987977","Type":"ContainerDied","Data":"88b72345e2258241c9ed0bb8e8b2bb93dc74bfd60bfe078eb16ad0948b5fbda6"}
Nov 03 21:37:15.606085 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.605890    4274 scope.go:122] "RemoveContainer" containerID="906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"
Nov 03 21:37:15.614249 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.614228    4274 scope.go:122] "RemoveContainer" containerID="906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"
Nov 03 21:37:15.614489 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:37:15.614468    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21\": container with ID starting with 906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21 not found: ID does not exist" containerID="906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"
Nov 03 21:37:15.614549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.614499    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21"} err="failed to get container status \"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21\": rpc error: code = NotFound desc = could not find container \"906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21\": container with ID starting with 906215f3a8f0072612e03cd3333ed31f07ba410296ec9ac37ea9b452b8222b21 not found: ID does not exist"
Nov 03 21:37:15.666136 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666109    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666275 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666224    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666370 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666271    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666432    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\" (UniqueName: \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666513 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666480    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666575 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666550    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\") pod \"ca5e335c-3073-43af-a9f8-36358f987977\" (UID: \"ca5e335c-3073-43af-a9f8-36358f987977\") "
Nov 03 21:37:15.666653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666627    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:15.666790 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666767    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:15.666887 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666869    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:15.666957 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666938    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:15.667021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.666977    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:15.669221 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.669195    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:15.674156 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.674130    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:37:15.674349 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.674322    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf" pod "ca5e335c-3073-43af-a9f8-36358f987977" (UID: "ca5e335c-3073-43af-a9f8-36358f987977"). InnerVolumeSpecName "kube-api-access-kbnbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:15.767538 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.767506    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:15.767538 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.767539    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e335c-3073-43af-a9f8-36358f987977-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:15.767735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.767550    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbnbf\" (UniqueName: \"kubernetes.io/projected/ca5e335c-3073-43af-a9f8-36358f987977-kube-api-access-kbnbf\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:15.767735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:15.767559    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e335c-3073-43af-a9f8-36358f987977-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:16.018338 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:16.018305    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:37:16.022425 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:16.022400    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7b11dd/simulator-single-main-7b11dd-kserve-7bdfb7c8d9-scw8p"]
Nov 03 21:37:17.212511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:17.212478    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5e335c-3073-43af-a9f8-36358f987977" path="/var/lib/kubelet/pods/ca5e335c-3073-43af-a9f8-36358f987977/volumes"
Nov 03 21:37:22.208572 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:22.208540    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:37:22.208950 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:22.208725    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy" containerID="cri-o://a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af" gracePeriod=30
Nov 03 21:37:27.366777 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:37:27.366736    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:37:27.468209 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.468179    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:37:27.573863 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.573835    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.573881    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.573911    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.573930    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.573956    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574063 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574018    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574073    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574124    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574173    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\") pod \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\" (UID: \"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a\") "
Nov 03 21:37:27.574331 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574288    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:27.574553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574373    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:27.574553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574392    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:37:27.574553 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574509    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:27.574700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574608    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:27.574700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574625    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.574700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574645    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.574700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574660    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.574700 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.574673    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.576202 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.576174    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:27.576319 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.576280    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:37:27.576373 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.576338    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:27.576481 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.576458    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74" pod "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" (UID: "3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a"). InnerVolumeSpecName "kube-api-access-lbg74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:27.662314 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.662272    4274 generic.go:356] "Generic (PLEG): container finished" podID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerID="a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af" exitCode=0
Nov 03 21:37:27.662446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.662338    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"
Nov 03 21:37:27.662446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.662356    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerDied","Data":"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"}
Nov 03 21:37:27.662446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.662413    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j" event={"ID":"3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a","Type":"ContainerDied","Data":"c03c9fd1a3ef74e6244e834f25299a94ae030ff3410d779d23dea92b3b771eb0"}
Nov 03 21:37:27.662446 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.662436    4274 scope.go:122] "RemoveContainer" containerID="a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"
Nov 03 21:37:27.671301 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.671283    4274 scope.go:122] "RemoveContainer" containerID="43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"
Nov 03 21:37:27.675751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.675728    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.675816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.675756    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.675816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.675771    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbg74\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-kube-api-access-lbg74\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.675816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.675785    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.675816 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.675800    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:27.678793 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.678779    4274 scope.go:122] "RemoveContainer" containerID="a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"
Nov 03 21:37:27.679050 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:37:27.679016    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af\": container with ID starting with a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af not found: ID does not exist" containerID="a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"
Nov 03 21:37:27.679094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.679060    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af"} err="failed to get container status \"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af\": rpc error: code = NotFound desc = could not find container \"a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af\": container with ID starting with a1c8bd16688382994b47120675986cc31abb292ee5e7e12f45ebdec49ca9b5af not found: ID does not exist"
Nov 03 21:37:27.679094 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.679084    4274 scope.go:122] "RemoveContainer" containerID="43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"
Nov 03 21:37:27.679336 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:37:27.679318    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8\": container with ID starting with 43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8 not found: ID does not exist" containerID="43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"
Nov 03 21:37:27.679393 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.679345    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8"} err="failed to get container status \"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8\": rpc error: code = NotFound desc = could not find container \"43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8\": container with ID starting with 43ccb2e731039026294a533b7a46433ec7936c44b02087b1b56b6fd894d686b8 not found: ID does not exist"
Nov 03 21:37:27.815864 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.815830    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:37:27.821022 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:27.820993    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7b11dd-openshift-default-685f4db597w8l5j"]
Nov 03 21:37:29.211585 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:29.211550    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" path="/var/lib/kubelet/pods/3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a/volumes"
Nov 03 21:37:40.570199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570162    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:37:40.570601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570557    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570577    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570590    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570595    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570604    4274 cpu_manager.go:414] "containerMap: removing container" podUID="ca5e335c-3073-43af-a9f8-36358f987977" containerName="main"
Nov 03 21:37:40.570736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570610    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5e335c-3073-43af-a9f8-36358f987977" containerName="main"
Nov 03 21:37:40.570736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570692    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570701    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a2d5c-47f2-4d0f-9bfb-6d8090f31d4a" containerName="istio-proxy"
Nov 03 21:37:40.570736 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.570707    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5e335c-3073-43af-a9f8-36358f987977" containerName="main"
Nov 03 21:37:40.575292 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.575272    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.670031 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.670002    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f43a5fbf-kserve-self-signed-certs\""
Nov 03 21:37:40.671912 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.671892    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:37:40.692529 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692501    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.692655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692534    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.692655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692560    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.692655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692629    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.692769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692754    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.692807 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.692777    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhb5\" (UniqueName: \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793459 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793424    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793464    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793517    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793551    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793620    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.793653 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793642    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhb5\" (UniqueName: \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.794089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.793950    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.794089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.794004    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.794089 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.794065    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.795833 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.795806    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.795988 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.795972    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.853915 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.853861    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhb5\" (UniqueName: \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\") pod \"e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") " pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:40.886292 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:40.886267    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:41.249763 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:41.249739    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:37:41.252411 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:37:41.252383    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4853ec87_0eb8_4d7c_ab89_e54e88b51fa1.slice/crio-80aa9fe3ab50c26a2510d7da29f0f82140f7eefaffa2b032fe7e4e8b037baecb WatchSource:0}: Error finding container 80aa9fe3ab50c26a2510d7da29f0f82140f7eefaffa2b032fe7e4e8b037baecb: Status 404 returned error can't find the container with id 80aa9fe3ab50c26a2510d7da29f0f82140f7eefaffa2b032fe7e4e8b037baecb
Nov 03 21:37:41.730817 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:41.730784    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" event={"ID":"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1","Type":"ContainerStarted","Data":"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"}
Nov 03 21:37:41.731218 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:41.730825    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" event={"ID":"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1","Type":"ContainerStarted","Data":"80aa9fe3ab50c26a2510d7da29f0f82140f7eefaffa2b032fe7e4e8b037baecb"}
Nov 03 21:37:41.840306 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:41.840262    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" podStartSLOduration=1.8402495239999999 podStartE2EDuration="1.840249524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:41.839900523 +0000 UTC m=+4086.766470759" watchObservedRunningTime="2026-11-03 21:37:41.840249524 +0000 UTC m=+4086.766819806"
Nov 03 21:37:41.886717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:41.886689    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:50.887449 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:50.887374    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:50.899737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:50.899714    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:51.801270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:37:51.801235    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:37:57.364228 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:37:57.364182    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:38:06.930270 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.930235    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:38:06.933865 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.933848    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.984701 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.984675    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdzgq\" (UniqueName: \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.984855 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.984718    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.984855 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.984743    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.984974 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.984940    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.985026 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.984995    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:06.985097 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:06.985076    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.028350 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.028322    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f43a5fbf-kserve-self-signed-certs\""
Nov 03 21:38:07.030120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.030088    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:38:07.086769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086732    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.086946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086793    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.086946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086823    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.086946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086893    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdzgq\" (UniqueName: \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.087174 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086945    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.087174 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.086988    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.087526 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.087489    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.087732 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.087593    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.087971 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.087921    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.089885 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.089856    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.090015 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.089994    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.145100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.145068    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdzgq\" (UniqueName: \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\") pod \"e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") " pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.246190 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.246119    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:07.609794 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.609770    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:38:07.612161 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:38:07.612126    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda93e08dc_b766_4ebe_bcd4_54459fa015b2.slice/crio-b9648129075e7c430594863dffba3d323ff71649c90ba566c6479ae7bb21c469 WatchSource:0}: Error finding container b9648129075e7c430594863dffba3d323ff71649c90ba566c6479ae7bb21c469: Status 404 returned error can't find the container with id b9648129075e7c430594863dffba3d323ff71649c90ba566c6479ae7bb21c469
Nov 03 21:38:07.868342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.868256    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" event={"ID":"a93e08dc-b766-4ebe-bcd4-54459fa015b2","Type":"ContainerStarted","Data":"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"}
Nov 03 21:38:07.868342 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.868294    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" event={"ID":"a93e08dc-b766-4ebe-bcd4-54459fa015b2","Type":"ContainerStarted","Data":"b9648129075e7c430594863dffba3d323ff71649c90ba566c6479ae7bb21c469"}
Nov 03 21:38:07.978770 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:07.978720    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" podStartSLOduration=1.978703968 podStartE2EDuration="1.978703968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:07.976294841 +0000 UTC m=+4112.902865077" watchObservedRunningTime="2026-11-03 21:38:07.978703968 +0000 UTC m=+4112.905274204"
Nov 03 21:38:08.246344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:08.246273    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:17.247219 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:17.247188    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:17.259841 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:17.259813    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:17.941400 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:17.941370    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:38:27.364726 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:38:27.364683    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:38:54.785181 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:38:54.785146    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:38:57.365797 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:38:57.365756    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:39:07.925470 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:07.925434    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:39:09.804835 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:09.804796    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" podUID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" containerName="main" containerID="cri-o://b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e" gracePeriod=45
Nov 03 21:39:10.048764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.048738    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:39:10.078211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078163    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078200    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078348 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078276    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078348 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078309    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\" (UniqueName: \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078360    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078468 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078382    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\") pod \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\" (UID: \"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1\") "
Nov 03 21:39:10.078578 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078540    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:10.078765 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078661    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:10.078894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078755    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:10.078894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078807    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.078894 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.078827    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.080420 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.080396    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "kube-api-access-8bhb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:10.080420 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.080400    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:10.080574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.080404    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm" pod "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" (UID: "4853ec87-0eb8-4d7c-ab89-e54e88b51fa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:10.179316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.179289    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.179316 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.179313    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhb5\" (UniqueName: \"kubernetes.io/projected/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-kube-api-access-8bhb5\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.179482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.179324    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.179482 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.179332    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:10.223923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.223888    4274 generic.go:356] "Generic (PLEG): container finished" podID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" containerID="b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e" exitCode=0
Nov 03 21:39:10.224088 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.223954    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"
Nov 03 21:39:10.224088 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.223971    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" event={"ID":"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1","Type":"ContainerDied","Data":"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"}
Nov 03 21:39:10.224088 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.224012    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf" event={"ID":"4853ec87-0eb8-4d7c-ab89-e54e88b51fa1","Type":"ContainerDied","Data":"80aa9fe3ab50c26a2510d7da29f0f82140f7eefaffa2b032fe7e4e8b037baecb"}
Nov 03 21:39:10.224088 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.224029    4274 scope.go:122] "RemoveContainer" containerID="b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"
Nov 03 21:39:10.233858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.233840    4274 scope.go:122] "RemoveContainer" containerID="b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"
Nov 03 21:39:10.234131 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:39:10.234110    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e\": container with ID starting with b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e not found: ID does not exist" containerID="b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"
Nov 03 21:39:10.234220 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.234142    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e"} err="failed to get container status \"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e\": rpc error: code = NotFound desc = could not find container \"b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e\": container with ID starting with b6879336bd3ad234b57bf4eb2f5388dadb4fb38e71d7cd9fe765f50f1d34250e not found: ID does not exist"
Nov 03 21:39:10.380424 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.380358    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:39:10.383430 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:10.383407    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f43a5fbf-kserve-7fb58f4c4f-pt8tf"]
Nov 03 21:39:11.212361 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:11.212329    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" path="/var/lib/kubelet/pods/4853ec87-0eb8-4d7c-ab89-e54e88b51fa1/volumes"
Nov 03 21:39:22.944994 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:22.944901    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" podUID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" containerName="main" containerID="cri-o://bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f" gracePeriod=45
Nov 03 21:39:23.193731 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.193701    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:39:23.294444 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294422    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294507    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294592 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294540    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294710 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294605    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294710 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294648    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294710 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294683    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\" (UniqueName: \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\") pod \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\" (UID: \"a93e08dc-b766-4ebe-bcd4-54459fa015b2\") "
Nov 03 21:39:23.294955 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294926    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:23.295025 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.294987    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:23.295277 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.295249    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:23.296574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.296553    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:23.296920 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.296898    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:23.296999 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.296903    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq" pod "a93e08dc-b766-4ebe-bcd4-54459fa015b2" (UID: "a93e08dc-b766-4ebe-bcd4-54459fa015b2"). InnerVolumeSpecName "kube-api-access-tdzgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:23.298231 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.298209    4274 generic.go:356] "Generic (PLEG): container finished" podID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" containerID="bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f" exitCode=0
Nov 03 21:39:23.298335 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.298248    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"
Nov 03 21:39:23.298399 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.298249    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" event={"ID":"a93e08dc-b766-4ebe-bcd4-54459fa015b2","Type":"ContainerDied","Data":"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"}
Nov 03 21:39:23.298399 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.298384    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k" event={"ID":"a93e08dc-b766-4ebe-bcd4-54459fa015b2","Type":"ContainerDied","Data":"b9648129075e7c430594863dffba3d323ff71649c90ba566c6479ae7bb21c469"}
Nov 03 21:39:23.298507 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.298415    4274 scope.go:122] "RemoveContainer" containerID="bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"
Nov 03 21:39:23.316641 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.316609    4274 scope.go:122] "RemoveContainer" containerID="bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"
Nov 03 21:39:23.316944 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:39:23.316924    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f\": container with ID starting with bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f not found: ID does not exist" containerID="bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"
Nov 03 21:39:23.317060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.316950    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f"} err="failed to get container status \"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f\": rpc error: code = NotFound desc = could not find container \"bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f\": container with ID starting with bd8aa98e21fff237eb80559d6d731fbe4779d867c21e0b1ed8b3c4d56a0eea3f not found: ID does not exist"
Nov 03 21:39:23.395924 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395897    4274 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-home\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.395924 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395920    4274 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tmp-dir\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.396116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395931    4274 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-model-cache\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.396116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395940    4274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93e08dc-b766-4ebe-bcd4-54459fa015b2-tls-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.396116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395950    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdzgq\" (UniqueName: \"kubernetes.io/projected/a93e08dc-b766-4ebe-bcd4-54459fa015b2-kube-api-access-tdzgq\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.396116 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.395959    4274 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93e08dc-b766-4ebe-bcd4-54459fa015b2-dshm\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:23.455994 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.455964    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:39:23.460073 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:23.460046    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f43a5fbf-kserve-76bb8569-sl89k"]
Nov 03 21:39:25.212436 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:25.212399    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" path="/var/lib/kubelet/pods/a93e08dc-b766-4ebe-bcd4-54459fa015b2/volumes"
Nov 03 21:39:27.549641 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:39:27.549601    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:39:54.982370 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982337    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"]
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982706    4274 cpu_manager.go:414] "containerMap: removing container" podUID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" containerName="main"
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982717    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" containerName="main"
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982735    4274 cpu_manager.go:414] "containerMap: removing container" podUID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" containerName="main"
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982740    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" containerName="main"
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982801    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4853ec87-0eb8-4d7c-ab89-e54e88b51fa1" containerName="main"
Nov 03 21:39:54.982866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.982812    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93e08dc-b766-4ebe-bcd4-54459fa015b2" containerName="main"
Nov 03 21:39:54.985786 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:54.985767    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.083683 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.083646    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"]
Nov 03 21:39:55.114522 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.114489    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-maas-api-tls\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.114685 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.114533    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-metrics-certs\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.114685 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.114569    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fee4dbd-a32e-40e3-9e6a-20296a51776a-openshift-service-ca\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.114781 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.114742    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fj4z\" (UniqueName: \"kubernetes.io/projected/9fee4dbd-a32e-40e3-9e6a-20296a51776a-kube-api-access-4fj4z\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.216208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.216178    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-maas-api-tls\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.216393 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.216220    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-metrics-certs\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.216393 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.216258    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fee4dbd-a32e-40e3-9e6a-20296a51776a-openshift-service-ca\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.216519 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.216486    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fj4z\" (UniqueName: \"kubernetes.io/projected/9fee4dbd-a32e-40e3-9e6a-20296a51776a-kube-api-access-4fj4z\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.216969 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.216948    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fee4dbd-a32e-40e3-9e6a-20296a51776a-openshift-service-ca\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.218620 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.218598    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-metrics-certs\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.218724 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.218705    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fee4dbd-a32e-40e3-9e6a-20296a51776a-maas-api-tls\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.273702 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.273672    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fj4z\" (UniqueName: \"kubernetes.io/projected/9fee4dbd-a32e-40e3-9e6a-20296a51776a-kube-api-access-4fj4z\") pod \"maas-api-bbfc75f54-5ptcg\" (UID: \"9fee4dbd-a32e-40e3-9e6a-20296a51776a\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.300266 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.300243    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:55.665477 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.665448    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"]
Nov 03 21:39:55.667483 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:39:55.667454    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fee4dbd_a32e_40e3_9e6a_20296a51776a.slice/crio-cf1d4d009aa6980558e54a15f0fb2624b45cfd1fa281dc97359d5276b3598b36 WatchSource:0}: Error finding container cf1d4d009aa6980558e54a15f0fb2624b45cfd1fa281dc97359d5276b3598b36: Status 404 returned error can't find the container with id cf1d4d009aa6980558e54a15f0fb2624b45cfd1fa281dc97359d5276b3598b36
Nov 03 21:39:55.668934 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:55.668916    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:39:56.500780 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:56.500738    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg" event={"ID":"9fee4dbd-a32e-40e3-9e6a-20296a51776a","Type":"ContainerStarted","Data":"cf1d4d009aa6980558e54a15f0fb2624b45cfd1fa281dc97359d5276b3598b36"}
Nov 03 21:39:57.369785 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:39:57.369737    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:39:57.507346 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:57.507310    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg" event={"ID":"9fee4dbd-a32e-40e3-9e6a-20296a51776a","Type":"ContainerStarted","Data":"fb0dbe2bdafc7e88ac3ed56e355ea480482c118fd12659a1b3a7b2c7805d4653"}
Nov 03 21:39:57.507722 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:57.507452    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:39:57.618626 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:39:57.618566    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg" podStartSLOduration=2.677081933 podStartE2EDuration="3.618544694s" totalImagesPullingTime="941.462761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:57.616726309 +0000 UTC m=+4222.543296546" watchObservedRunningTime="2026-11-03 21:39:57.618544694 +0000 UTC m=+4222.545114960"
Nov 03 21:40:03.516474 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:03.516444    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-5ptcg"
Nov 03 21:40:03.673882 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:03.673833    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 21:40:03.674231 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:03.674181    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" podUID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" containerName="maas-api" containerID="cri-o://95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd" gracePeriod=30
Nov 03 21:40:04.921407 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:04.921381    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 21:40:05.003434 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.003406    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\") pod \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") "
Nov 03 21:40:05.003601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.003443    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\" (UniqueName: \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\") pod \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") "
Nov 03 21:40:05.003601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.003493    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\") pod \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") "
Nov 03 21:40:05.003601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.003552    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\") pod \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\" (UID: \"3b2f12b1-2087-4bb2-92d5-126e0d154b0f\") "
Nov 03 21:40:05.003961 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.003929    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca" pod "3b2f12b1-2087-4bb2-92d5-126e0d154b0f" (UID: "3b2f12b1-2087-4bb2-92d5-126e0d154b0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:05.005528 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.005507    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb" pod "3b2f12b1-2087-4bb2-92d5-126e0d154b0f" (UID: "3b2f12b1-2087-4bb2-92d5-126e0d154b0f"). InnerVolumeSpecName "kube-api-access-nnmdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:05.005528 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.005516    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls" pod "3b2f12b1-2087-4bb2-92d5-126e0d154b0f" (UID: "3b2f12b1-2087-4bb2-92d5-126e0d154b0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:40:05.005651 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.005599    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs" pod "3b2f12b1-2087-4bb2-92d5-126e0d154b0f" (UID: "3b2f12b1-2087-4bb2-92d5-126e0d154b0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:40:05.105299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.105272    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:05.105299 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.105297    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:05.105470 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.105307    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnmdb\" (UniqueName: \"kubernetes.io/projected/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-kube-api-access-nnmdb\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:05.105470 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.105316    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b2f12b1-2087-4bb2-92d5-126e0d154b0f-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:05.545547 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.545515    4274 generic.go:356] "Generic (PLEG): container finished" podID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" containerID="95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd" exitCode=0
Nov 03 21:40:05.545734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.545564    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"
Nov 03 21:40:05.545734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.545564    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" event={"ID":"3b2f12b1-2087-4bb2-92d5-126e0d154b0f","Type":"ContainerDied","Data":"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"}
Nov 03 21:40:05.545734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.545686    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-bp7x7" event={"ID":"3b2f12b1-2087-4bb2-92d5-126e0d154b0f","Type":"ContainerDied","Data":"6897fd77084f26f1fa53958d818d3a2e43e2ace9a397e0ffb16935ee4b710ffc"}
Nov 03 21:40:05.545734 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.545709    4274 scope.go:122] "RemoveContainer" containerID="95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"
Nov 03 21:40:05.554783 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.554766    4274 scope.go:122] "RemoveContainer" containerID="95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"
Nov 03 21:40:05.555028 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:40:05.555010    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd\": container with ID starting with 95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd not found: ID does not exist" containerID="95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"
Nov 03 21:40:05.555097 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.555058    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd"} err="failed to get container status \"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd\": rpc error: code = NotFound desc = could not find container \"95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd\": container with ID starting with 95ee5cb8e70aba54f9383660c23b692616fb989be5777c46f6da835bb53af0cd not found: ID does not exist"
Nov 03 21:40:05.697977 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.697946    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 21:40:05.744030 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:05.744005    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-bp7x7"]
Nov 03 21:40:07.212242 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:07.212212    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" path="/var/lib/kubelet/pods/3b2f12b1-2087-4bb2-92d5-126e0d154b0f/volumes"
Nov 03 21:40:09.389855 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.389820    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:40:09.390265 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.390212    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" containerName="maas-api"
Nov 03 21:40:09.390265 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.390224    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" containerName="maas-api"
Nov 03 21:40:09.390335 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.390289    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2f12b1-2087-4bb2-92d5-126e0d154b0f" containerName="maas-api"
Nov 03 21:40:09.394709 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.394685    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.445491 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.445458    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4f1680df\""
Nov 03 21:40:09.445643 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.445617    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.445756 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.445736    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggrd\" (UniqueName: \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.491725 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.491694    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4f1680df-dockercfg-7fvvz\""
Nov 03 21:40:09.494695 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.494643    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:40:09.546463 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.546429    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggrd\" (UniqueName: \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.546604 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.546556    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.547330 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.547310    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.560046 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.560000    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:40:09.563992 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.563971    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.603761 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.603736    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggrd\" (UniqueName: \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\") pod \"payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.619236 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.619214    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:40:09.647748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.647690    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52sx\" (UniqueName: \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.647845 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.647814    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.707562 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.707536    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:09.748547 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.748515    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52sx\" (UniqueName: \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.748681 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.748604    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.749258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.749239    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.806484 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.806449    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52sx\" (UniqueName: \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\") pod \"payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") " pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:09.877333 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:09.877291    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:10.069256 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.069228    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:40:10.071480 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:40:10.071449    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9adfb2_4846_4094_a37e_17bf2f47e51a.slice/crio-f0e447552e69811745930e6b1e1b905b2c1da78bfa7ad04b9be8474da35f17ed WatchSource:0}: Error finding container f0e447552e69811745930e6b1e1b905b2c1da78bfa7ad04b9be8474da35f17ed: Status 404 returned error can't find the container with id f0e447552e69811745930e6b1e1b905b2c1da78bfa7ad04b9be8474da35f17ed
Nov 03 21:40:10.240477 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.240453    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:40:10.242776 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:40:10.242747    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9cc0e4_9223_466b_a765_c598169b92f6.slice/crio-bffb3bddb0b7c9ad67bf849d54d2e824ca988d5a1a9a82ac2ab19c44c0e43e29 WatchSource:0}: Error finding container bffb3bddb0b7c9ad67bf849d54d2e824ca988d5a1a9a82ac2ab19c44c0e43e29: Status 404 returned error can't find the container with id bffb3bddb0b7c9ad67bf849d54d2e824ca988d5a1a9a82ac2ab19c44c0e43e29
Nov 03 21:40:10.570531 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.570498    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" event={"ID":"3c9adfb2-4846-4094-a37e-17bf2f47e51a","Type":"ContainerStarted","Data":"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"}
Nov 03 21:40:10.570958 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.570547    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" event={"ID":"3c9adfb2-4846-4094-a37e-17bf2f47e51a","Type":"ContainerStarted","Data":"f0e447552e69811745930e6b1e1b905b2c1da78bfa7ad04b9be8474da35f17ed"}
Nov 03 21:40:10.570958 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.570602    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:10.572282 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.572259    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" event={"ID":"1c9cc0e4-9223-466b-a765-c598169b92f6","Type":"ContainerStarted","Data":"a0785b62b2ce7b170e9107b58bc805455d82f716710dd062fedaed8b83c78fdc"}
Nov 03 21:40:10.572398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.572299    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" event={"ID":"1c9cc0e4-9223-466b-a765-c598169b92f6","Type":"ContainerStarted","Data":"bffb3bddb0b7c9ad67bf849d54d2e824ca988d5a1a9a82ac2ab19c44c0e43e29"}
Nov 03 21:40:10.572398 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.572335    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:10.678399 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.678361    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" podStartSLOduration=1.678350409 podStartE2EDuration="1.678350409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:10.67744142 +0000 UTC m=+4235.604011658" watchObservedRunningTime="2026-11-03 21:40:10.678350409 +0000 UTC m=+4235.604920636"
Nov 03 21:40:10.786383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.786344    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" podStartSLOduration=1.7863349670000002 podStartE2EDuration="1.786334967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:10.784682274 +0000 UTC m=+4235.711252522" watchObservedRunningTime="2026-11-03 21:40:10.786334967 +0000 UTC m=+4235.712905217"
Nov 03 21:40:10.977676 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.977598    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:40:10.981100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:10.981079    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.029586 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.029562    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4f1680df-serving-cert\""
Nov 03 21:40:11.038549 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.038528    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:40:11.059408 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.059385    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9552\" (UniqueName: \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.059505 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.059468    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.059593 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.059570    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.059654 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.059639    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.160610 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.160580    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9552\" (UniqueName: \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.160776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.160672    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.160776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.160715    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.160776 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.160765    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.161259 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:40:11.161229    4274 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-serving-cert: secret "maas-api-e2e-ait-4f1680df-serving-cert" not found
Nov 03 21:40:11.161396 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:40:11.161302    4274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls podName:f9d8360e-e117-43bd-a443-888edb653873 nodeName:}" failed. No retries permitted until 2026-11-03 21:40:11.661283046 +0000 UTC m=+4236.587853262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls") pod "maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" (UID: "f9d8360e-e117-43bd-a443-888edb653873") : secret "maas-api-e2e-ait-4f1680df-serving-cert" not found
Nov 03 21:40:11.161570 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.161541    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.163651 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.163623    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.220262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.220233    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9552\" (UniqueName: \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.665827 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.665783    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.668001 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.667979    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") pod \"maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:11.894283 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:11.894252    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:12.259463 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:12.259438    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:40:12.261157 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:40:12.261123    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d8360e_e117_43bd_a443_888edb653873.slice/crio-b5eb925246e6b0215fdefd11f0adb0c79691fe0a5cbdfc9403e87907bdb9b070 WatchSource:0}: Error finding container b5eb925246e6b0215fdefd11f0adb0c79691fe0a5cbdfc9403e87907bdb9b070: Status 404 returned error can't find the container with id b5eb925246e6b0215fdefd11f0adb0c79691fe0a5cbdfc9403e87907bdb9b070
Nov 03 21:40:12.582384 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:12.582298    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" event={"ID":"f9d8360e-e117-43bd-a443-888edb653873","Type":"ContainerStarted","Data":"b5eb925246e6b0215fdefd11f0adb0c79691fe0a5cbdfc9403e87907bdb9b070"}
Nov 03 21:40:13.587934 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:13.587893    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" event={"ID":"f9d8360e-e117-43bd-a443-888edb653873","Type":"ContainerStarted","Data":"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"}
Nov 03 21:40:13.588337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:13.587969    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:13.698551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:13.698437    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" podStartSLOduration=2.799625541 podStartE2EDuration="3.69841901s" totalImagesPullingTime="898.793469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:13.696989526 +0000 UTC m=+4238.623559775" watchObservedRunningTime="2026-11-03 21:40:13.69841901 +0000 UTC m=+4238.624989260"
Nov 03 21:40:19.597735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:19.597703    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:40:21.579100 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:21.579074    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:40:21.579476 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:40:21.579143    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:40:27.364916 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:40:27.364881    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:40:57.364445 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:40:57.364405    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:41:03.640978 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:03.640943    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:41:03.750852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:03.750817    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:41:08.641852 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.641794    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" podUID="1c9cc0e4-9223-466b-a765-c598169b92f6" containerName="payload-processing" containerID="cri-o://a0785b62b2ce7b170e9107b58bc805455d82f716710dd062fedaed8b83c78fdc" gracePeriod=25
Nov 03 21:41:08.751233 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.751183    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" podUID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" containerName="payload-pre-processing" containerID="cri-o://4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147" gracePeriod=25
Nov 03 21:41:08.838746 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.838713    4274 generic.go:356] "Generic (PLEG): container finished" podID="1c9cc0e4-9223-466b-a765-c598169b92f6" containerID="a0785b62b2ce7b170e9107b58bc805455d82f716710dd062fedaed8b83c78fdc" exitCode=0
Nov 03 21:41:08.838906 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.838812    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" event={"ID":"1c9cc0e4-9223-466b-a765-c598169b92f6","Type":"ContainerDied","Data":"a0785b62b2ce7b170e9107b58bc805455d82f716710dd062fedaed8b83c78fdc"}
Nov 03 21:41:08.905083 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.905001    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:41:08.956923 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.956892    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\" (UniqueName: \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\") pod \"1c9cc0e4-9223-466b-a765-c598169b92f6\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") "
Nov 03 21:41:08.957110 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.957070    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\") pod \"1c9cc0e4-9223-466b-a765-c598169b92f6\" (UID: \"1c9cc0e4-9223-466b-a765-c598169b92f6\") "
Nov 03 21:41:08.957538 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.957496    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume" pod "1c9cc0e4-9223-466b-a765-c598169b92f6" (UID: "1c9cc0e4-9223-466b-a765-c598169b92f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:08.959223 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.959188    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx" pod "1c9cc0e4-9223-466b-a765-c598169b92f6" (UID: "1c9cc0e4-9223-466b-a765-c598169b92f6"). InnerVolumeSpecName "kube-api-access-s52sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:08.999975 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:08.999954    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:41:09.058601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.058572    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\" (UniqueName: \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\") pod \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") "
Nov 03 21:41:09.058756 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.058622    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\") pod \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\" (UID: \"3c9adfb2-4846-4094-a37e-17bf2f47e51a\") "
Nov 03 21:41:09.058903 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.058889    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9cc0e4-9223-466b-a765-c598169b92f6-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.058976 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.058903    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52sx\" (UniqueName: \"kubernetes.io/projected/1c9cc0e4-9223-466b-a765-c598169b92f6-kube-api-access-s52sx\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.059140 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.059108    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume" pod "3c9adfb2-4846-4094-a37e-17bf2f47e51a" (UID: "3c9adfb2-4846-4094-a37e-17bf2f47e51a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:09.060685 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.060663    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd" pod "3c9adfb2-4846-4094-a37e-17bf2f47e51a" (UID: "3c9adfb2-4846-4094-a37e-17bf2f47e51a"). InnerVolumeSpecName "kube-api-access-cggrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:09.160426 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.160349    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggrd\" (UniqueName: \"kubernetes.io/projected/3c9adfb2-4846-4094-a37e-17bf2f47e51a-kube-api-access-cggrd\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.160426 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.160382    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9adfb2-4846-4094-a37e-17bf2f47e51a-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:09.845107 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.845077    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"
Nov 03 21:41:09.845107 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.845095    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh" event={"ID":"1c9cc0e4-9223-466b-a765-c598169b92f6","Type":"ContainerDied","Data":"bffb3bddb0b7c9ad67bf849d54d2e824ca988d5a1a9a82ac2ab19c44c0e43e29"}
Nov 03 21:41:09.845646 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.845158    4274 scope.go:122] "RemoveContainer" containerID="a0785b62b2ce7b170e9107b58bc805455d82f716710dd062fedaed8b83c78fdc"
Nov 03 21:41:09.847201 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.847169    4274 generic.go:356] "Generic (PLEG): container finished" podID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" containerID="4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147" exitCode=0
Nov 03 21:41:09.847394 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.847230    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" event={"ID":"3c9adfb2-4846-4094-a37e-17bf2f47e51a","Type":"ContainerDied","Data":"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"}
Nov 03 21:41:09.847394 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.847246    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"
Nov 03 21:41:09.847394 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.847266    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj" event={"ID":"3c9adfb2-4846-4094-a37e-17bf2f47e51a","Type":"ContainerDied","Data":"f0e447552e69811745930e6b1e1b905b2c1da78bfa7ad04b9be8474da35f17ed"}
Nov 03 21:41:09.853552 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.853534    4274 scope.go:122] "RemoveContainer" containerID="4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"
Nov 03 21:41:09.860862 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.860841    4274 scope.go:122] "RemoveContainer" containerID="4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"
Nov 03 21:41:09.861145 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:41:09.861118    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147\": container with ID starting with 4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147 not found: ID does not exist" containerID="4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"
Nov 03 21:41:09.861211 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.861154    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147"} err="failed to get container status \"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147\": rpc error: code = NotFound desc = could not find container \"4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147\": container with ID starting with 4fb7c0bbf0a1f04a18e84b0cc356e9dc936614dfcee4558c4c9bcdfc22a5b147 not found: ID does not exist"
Nov 03 21:41:09.998015 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:09.997958    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:41:10.044186 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:10.044157    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4f1680df-5495f964c-js2hj"]
Nov 03 21:41:10.103605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:10.103579    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:41:10.107364 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:10.107340    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4f1680df-6c7fb59769-s6fzh"]
Nov 03 21:41:11.212785 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:11.212754    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9cc0e4-9223-466b-a765-c598169b92f6" path="/var/lib/kubelet/pods/1c9cc0e4-9223-466b-a765-c598169b92f6/volumes"
Nov 03 21:41:11.213195 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:11.213135    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" path="/var/lib/kubelet/pods/3c9adfb2-4846-4094-a37e-17bf2f47e51a/volumes"
Nov 03 21:41:11.887265 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:11.887234    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:41:11.887448 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:11.887418    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" podUID="f9d8360e-e117-43bd-a443-888edb653873" containerName="maas-api" containerID="cri-o://2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963" gracePeriod=30
Nov 03 21:41:12.133471 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.133450    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:41:12.187391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.187340    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\" (UniqueName: \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\") pod \"f9d8360e-e117-43bd-a443-888edb653873\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") "
Nov 03 21:41:12.187391 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.187386    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") pod \"f9d8360e-e117-43bd-a443-888edb653873\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") "
Nov 03 21:41:12.187577 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.187453    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\") pod \"f9d8360e-e117-43bd-a443-888edb653873\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") "
Nov 03 21:41:12.187577 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.187471    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\") pod \"f9d8360e-e117-43bd-a443-888edb653873\" (UID: \"f9d8360e-e117-43bd-a443-888edb653873\") "
Nov 03 21:41:12.187881 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.187854    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca" pod "f9d8360e-e117-43bd-a443-888edb653873" (UID: "f9d8360e-e117-43bd-a443-888edb653873"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:12.189443 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.189420    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls" pod "f9d8360e-e117-43bd-a443-888edb653873" (UID: "f9d8360e-e117-43bd-a443-888edb653873"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:41:12.189557 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.189449    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs" pod "f9d8360e-e117-43bd-a443-888edb653873" (UID: "f9d8360e-e117-43bd-a443-888edb653873"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:41:12.189615 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.189564    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552" pod "f9d8360e-e117-43bd-a443-888edb653873" (UID: "f9d8360e-e117-43bd-a443-888edb653873"). InnerVolumeSpecName "kube-api-access-b9552". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:12.288542 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.288498    4274 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-metrics-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.288542 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.288532    4274 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8360e-e117-43bd-a443-888edb653873-openshift-service-ca\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.288542 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.288547    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9552\" (UniqueName: \"kubernetes.io/projected/f9d8360e-e117-43bd-a443-888edb653873-kube-api-access-b9552\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.289058 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.288563    4274 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d8360e-e117-43bd-a443-888edb653873-maas-api-tls\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:12.868685 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.868620    4274 generic.go:356] "Generic (PLEG): container finished" podID="f9d8360e-e117-43bd-a443-888edb653873" containerID="2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963" exitCode=0
Nov 03 21:41:12.868869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.868681    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" event={"ID":"f9d8360e-e117-43bd-a443-888edb653873","Type":"ContainerDied","Data":"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"}
Nov 03 21:41:12.868869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.868760    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6" event={"ID":"f9d8360e-e117-43bd-a443-888edb653873","Type":"ContainerDied","Data":"b5eb925246e6b0215fdefd11f0adb0c79691fe0a5cbdfc9403e87907bdb9b070"}
Nov 03 21:41:12.868869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.868690    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"
Nov 03 21:41:12.868869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.868809    4274 scope.go:122] "RemoveContainer" containerID="2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"
Nov 03 21:41:12.878804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.878782    4274 scope.go:122] "RemoveContainer" containerID="2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"
Nov 03 21:41:12.879092 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:41:12.879072    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963\": container with ID starting with 2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963 not found: ID does not exist" containerID="2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"
Nov 03 21:41:12.879168 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:12.879103    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963"} err="failed to get container status \"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963\": rpc error: code = NotFound desc = could not find container \"2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963\": container with ID starting with 2dc0939a42abaeead2c7889a241c4d375d6d07f7ed7d6246169295adf2cc2963 not found: ID does not exist"
Nov 03 21:41:13.021348 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:13.021317    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:41:13.067335 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:13.067306    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f1680df-677f5d8d58-rb7f6"]
Nov 03 21:41:13.212121 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:13.212055    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d8360e-e117-43bd-a443-888edb653873" path="/var/lib/kubelet/pods/f9d8360e-e117-43bd-a443-888edb653873/volumes"
Nov 03 21:41:20.174174 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174141    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:41:20.174539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174490    4274 cpu_manager.go:414] "containerMap: removing container" podUID="f9d8360e-e117-43bd-a443-888edb653873" containerName="maas-api"
Nov 03 21:41:20.174539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174501    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d8360e-e117-43bd-a443-888edb653873" containerName="maas-api"
Nov 03 21:41:20.174539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174516    4274 cpu_manager.go:414] "containerMap: removing container" podUID="1c9cc0e4-9223-466b-a765-c598169b92f6" containerName="payload-processing"
Nov 03 21:41:20.174539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174523    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9cc0e4-9223-466b-a765-c598169b92f6" containerName="payload-processing"
Nov 03 21:41:20.174539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174538    4274 cpu_manager.go:414] "containerMap: removing container" podUID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" containerName="payload-pre-processing"
Nov 03 21:41:20.174727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174544    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" containerName="payload-pre-processing"
Nov 03 21:41:20.174727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174617    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d8360e-e117-43bd-a443-888edb653873" containerName="maas-api"
Nov 03 21:41:20.174727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174626    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9cc0e4-9223-466b-a765-c598169b92f6" containerName="payload-processing"
Nov 03 21:41:20.174727 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.174634    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9adfb2-4846-4094-a37e-17bf2f47e51a" containerName="payload-pre-processing"
Nov 03 21:41:20.179069 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.179026    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.227869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.227836    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-bbda3a-openshift-default-dockercfg-mg67q\""
Nov 03 21:41:20.256252 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256226    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256364 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256264    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256364 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256344    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256436 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256414    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256497 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256480    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256536 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256508    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7xh\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256571 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256535    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256643 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256629    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.256682 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.256658    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.276960 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.276938    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:41:20.357134 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357103    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357146    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357164    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7xh\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357196    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357218    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357258 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357237    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357533 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357295    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357533 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357330    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357533 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357374    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357680 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357548    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357680 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357576    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.357993 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357969    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.358133 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.357969    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.358133 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.358056    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.359858 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.359830    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.359990 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.359910    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.414872 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.414841    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.414997 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.414906    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7xh\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\") pod \"e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") " pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.491725 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.491639    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:20.908555 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.908530    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:41:20.910892 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:41:20.910860    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904b5e58_d708_4645_8740_5682267e8f87.slice/crio-47d0ad3e23f358e7e3179b81200a40babdf1c1963cd3993063b4d9d3b633c3d3 WatchSource:0}: Error finding container 47d0ad3e23f358e7e3179b81200a40babdf1c1963cd3993063b4d9d3b633c3d3: Status 404 returned error can't find the container with id 47d0ad3e23f358e7e3179b81200a40babdf1c1963cd3993063b4d9d3b633c3d3
Nov 03 21:41:20.912978 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.912946    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:20.913084 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.913012    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:20.913084 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:20.913070    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:21.910601 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:21.910555    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerStarted","Data":"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"}
Nov 03 21:41:21.911097 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:21.910618    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerStarted","Data":"47d0ad3e23f358e7e3179b81200a40babdf1c1963cd3993063b4d9d3b633c3d3"}
Nov 03 21:41:22.069230 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:22.069185    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podStartSLOduration=2.069172507 podStartE2EDuration="2.069172507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:22.06868028 +0000 UTC m=+4306.995250517" watchObservedRunningTime="2026-11-03 21:41:22.069172507 +0000 UTC m=+4306.995742744"
Nov 03 21:41:22.492708 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:22.492670    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:22.497522 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:22.497497    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:22.915985 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:22.915952    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:22.916804 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:22.916785    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:27.365270 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:41:27.365235    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:41:36.573437 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.573358    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:41:36.578413 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.578395    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.627250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.627228    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-bbda3a-dockercfg-xp8cx\""
Nov 03 21:41:36.627250 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.627230    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-bbda3a\""
Nov 03 21:41:36.678744 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.678708    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:41:36.707860 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.707828    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnc4t\" (UniqueName: \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.708032 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.707967    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.809343 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.809304    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnc4t\" (UniqueName: \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.809503 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.809401    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.810064 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.810030    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.866655 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.866584    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnc4t\" (UniqueName: \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\") pod \"payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:36.891072 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:36.891029    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:37.255728 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:37.255701    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:41:37.258812 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:41:37.258784    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc631e1e6_9fb0_4e2e_b095_67584b492102.slice/crio-90e027b77a7318d160c154d1fdbbaaff2bb06d0015e599e5aea8245387cc513b WatchSource:0}: Error finding container 90e027b77a7318d160c154d1fdbbaaff2bb06d0015e599e5aea8245387cc513b: Status 404 returned error can't find the container with id 90e027b77a7318d160c154d1fdbbaaff2bb06d0015e599e5aea8245387cc513b
Nov 03 21:41:37.984145 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:37.984098    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" event={"ID":"c631e1e6-9fb0-4e2e-b095-67584b492102","Type":"ContainerStarted","Data":"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"}
Nov 03 21:41:37.984551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:37.984156    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" event={"ID":"c631e1e6-9fb0-4e2e-b095-67584b492102","Type":"ContainerStarted","Data":"90e027b77a7318d160c154d1fdbbaaff2bb06d0015e599e5aea8245387cc513b"}
Nov 03 21:41:37.984551 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:37.984179    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:38.092670 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:38.092578    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" podStartSLOduration=2.092564307 podStartE2EDuration="2.092564307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:38.091567476 +0000 UTC m=+4323.018137714" watchObservedRunningTime="2026-11-03 21:41:38.092564307 +0000 UTC m=+4323.019134544"
Nov 03 21:41:48.992869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:48.992842    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:41:53.958237 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:53.958187    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:41:53.958771 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:53.958268    4274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:41:56.068478 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.068391    4274 generic.go:356] "Generic (PLEG): container finished" podID="904b5e58-d708-4645-8740-5682267e8f87" containerID="9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968" exitCode=0
Nov 03 21:41:56.068888 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.068475    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerDied","Data":"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"}
Nov 03 21:41:56.068940 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.068896    4274 scope.go:122] "RemoveContainer" containerID="9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"
Nov 03 21:41:56.070097 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.070064    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:56.070199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.070149    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:56.070268 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:56.070198    4274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:57.075815 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:57.075782    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerStarted","Data":"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"}
Nov 03 21:41:57.390196 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:41:57.390118    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:41:57.492078 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:57.492017    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:41:57.492289 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:57.492268    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 21:41:57.492410 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:57.492324    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 21:41:58.492375 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:58.492339    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 21:41:58.492782 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:58.492396    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 21:41:59.492720 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:59.492680    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 21:41:59.493168 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:41:59.492737    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 21:42:00.492264 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:00.492229    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 21:42:00.492452 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:00.492309    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 21:42:01.492428 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:01.492390    4274 patch_prober.go:29] interesting pod/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 21:42:01.492850 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:01.492452    4274 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 21:42:02.496618 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:02.496590    4274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:42:02.497030 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:02.496863    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:42:02.497490 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:42:02.497474    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:42:27.364461 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:42:27.364422    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:42:57.365427 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:42:57.365387    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:43:27.364565 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:43:27.364524    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:43:29.648093 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:29.648058    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:43:34.648771 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.648702    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" podUID="c631e1e6-9fb0-4e2e-b095-67584b492102" containerName="payload-pre-processing" containerID="cri-o://6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a" gracePeriod=25
Nov 03 21:43:34.897504 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.897479    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:43:34.968189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.968113    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\") pod \"c631e1e6-9fb0-4e2e-b095-67584b492102\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") "
Nov 03 21:43:34.968189 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.968169    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\" (UniqueName: \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\") pod \"c631e1e6-9fb0-4e2e-b095-67584b492102\" (UID: \"c631e1e6-9fb0-4e2e-b095-67584b492102\") "
Nov 03 21:43:34.968487 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.968466    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume" pod "c631e1e6-9fb0-4e2e-b095-67584b492102" (UID: "c631e1e6-9fb0-4e2e-b095-67584b492102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:34.970247 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:34.970225    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t" pod "c631e1e6-9fb0-4e2e-b095-67584b492102" (UID: "c631e1e6-9fb0-4e2e-b095-67584b492102"). InnerVolumeSpecName "kube-api-access-xnc4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:35.069293 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.069258    4274 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c631e1e6-9fb0-4e2e-b095-67584b492102-plugins-config-volume\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:35.069293 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.069288    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnc4t\" (UniqueName: \"kubernetes.io/projected/c631e1e6-9fb0-4e2e-b095-67584b492102-kube-api-access-xnc4t\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:35.509605 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.509573    4274 generic.go:356] "Generic (PLEG): container finished" podID="c631e1e6-9fb0-4e2e-b095-67584b492102" containerID="6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a" exitCode=0
Nov 03 21:43:35.509764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.509612    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"
Nov 03 21:43:35.509764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.509653    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" event={"ID":"c631e1e6-9fb0-4e2e-b095-67584b492102","Type":"ContainerDied","Data":"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"}
Nov 03 21:43:35.509764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.509711    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj" event={"ID":"c631e1e6-9fb0-4e2e-b095-67584b492102","Type":"ContainerDied","Data":"90e027b77a7318d160c154d1fdbbaaff2bb06d0015e599e5aea8245387cc513b"}
Nov 03 21:43:35.509764 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.509733    4274 scope.go:122] "RemoveContainer" containerID="6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"
Nov 03 21:43:35.519280 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.519251    4274 scope.go:122] "RemoveContainer" containerID="6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"
Nov 03 21:43:35.519535 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:43:35.519517    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a\": container with ID starting with 6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a not found: ID does not exist" containerID="6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"
Nov 03 21:43:35.519606 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.519542    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a"} err="failed to get container status \"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a\": rpc error: code = NotFound desc = could not find container \"6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a\": container with ID starting with 6ff3f055a3e1b77aedfec0af76664eb009c0a6de20fb6a9acdb2ef7f6702f23a not found: ID does not exist"
Nov 03 21:43:35.662192 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.662165    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:43:35.668574 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:35.668549    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-bbda3a-5b7dcd6d95-hqcmj"]
Nov 03 21:43:37.211748 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:37.211714    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c631e1e6-9fb0-4e2e-b095-67584b492102" path="/var/lib/kubelet/pods/c631e1e6-9fb0-4e2e-b095-67584b492102/volumes"
Nov 03 21:43:48.574954 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:48.574872    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:43:48.575431 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:48.575094    4274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy" containerID="cri-o://0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139" gracePeriod=30
Nov 03 21:43:53.818543 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.818515    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:43:53.841639 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841586    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.841639 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841631    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.841769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841659    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.841769 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841742    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.841866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841796    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.841866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841831    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.842028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841909    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.842028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841986    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.842028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842015    4274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\") pod \"904b5e58-d708-4645-8740-5682267e8f87\" (UID: \"904b5e58-d708-4645-8740-5682267e8f87\") "
Nov 03 21:43:53.842028 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.841998    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:53.842276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842011    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:53.842276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842102    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:53.842276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842240    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:53.842276 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842259    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:53.842443 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842416    4274 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-credential-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.842443 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842434    4274 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-socket\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.842514 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842448    4274 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-data\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.842514 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842461    4274 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-workload-certs\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.842514 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.842474    4274 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904b5e58-d708-4645-8740-5682267e8f87-istiod-ca-cert\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.843989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.843970    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:43:53.844137 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.844120    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "kube-api-access-vw7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:53.844230 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.844209    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:53.844269 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.844225    4274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token" pod "904b5e58-d708-4645-8740-5682267e8f87" (UID: "904b5e58-d708-4645-8740-5682267e8f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:53.943704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.943675    4274 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904b5e58-d708-4645-8740-5682267e8f87-istio-podinfo\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.943704 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.943699    4274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7xh\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-kube-api-access-vw7xh\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.943869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.943711    4274 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904b5e58-d708-4645-8740-5682267e8f87-istio-token\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:53.943869 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:53.943719    4274 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904b5e58-d708-4645-8740-5682267e8f87-istio-envoy\") on node \"ip-10-0-128-118.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:54.598304 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.598268    4274 generic.go:356] "Generic (PLEG): container finished" podID="904b5e58-d708-4645-8740-5682267e8f87" containerID="0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139" exitCode=0
Nov 03 21:43:54.598511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.598317    4274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"
Nov 03 21:43:54.598511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.598343    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerDied","Data":"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"}
Nov 03 21:43:54.598511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.598395    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx" event={"ID":"904b5e58-d708-4645-8740-5682267e8f87","Type":"ContainerDied","Data":"47d0ad3e23f358e7e3179b81200a40babdf1c1963cd3993063b4d9d3b633c3d3"}
Nov 03 21:43:54.598511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.598422    4274 scope.go:122] "RemoveContainer" containerID="0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"
Nov 03 21:43:54.607449 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.607432    4274 scope.go:122] "RemoveContainer" containerID="9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"
Nov 03 21:43:54.614905 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.614888    4274 scope.go:122] "RemoveContainer" containerID="0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"
Nov 03 21:43:54.615263 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:43:54.615242    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139\": container with ID starting with 0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139 not found: ID does not exist" containerID="0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"
Nov 03 21:43:54.615344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.615275    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139"} err="failed to get container status \"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139\": rpc error: code = NotFound desc = could not find container \"0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139\": container with ID starting with 0923d986e0de9c250b2134de4bd3aa67d5376f343c08ead955173de0281e4139 not found: ID does not exist"
Nov 03 21:43:54.615344 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.615298    4274 scope.go:122] "RemoveContainer" containerID="9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"
Nov 03 21:43:54.615537 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:43:54.615518    4274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968\": container with ID starting with 9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968 not found: ID does not exist" containerID="9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"
Nov 03 21:43:54.615576 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.615543    4274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968"} err="failed to get container status \"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968\": rpc error: code = NotFound desc = could not find container \"9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968\": container with ID starting with 9bfcb80c9f63553e84a91572b0705ab8fa7ec79e1c3a55c413db1f9d04cf3968 not found: ID does not exist"
Nov 03 21:43:54.798196 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.798167    4274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:43:54.803663 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:54.803635    4274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-bbda3a-openshift-default-884d959df-6q9tx"]
Nov 03 21:43:55.211614 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:43:55.211582    4274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904b5e58-d708-4645-8740-5682267e8f87" path="/var/lib/kubelet/pods/904b5e58-d708-4645-8740-5682267e8f87/volumes"
Nov 03 21:43:57.376989 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:43:57.376933    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:44:27.365258 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:44:27.365223    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:44:57.364408 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:44:57.364363    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:45:00.831697 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:00.831665    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-784fb86d65-6kg8v_1a2f424c-7201-4beb-acc4-4481306fb5a3/authorino/0.log"
Nov 03 21:45:11.872703 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:11.872669    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-658cf6f9cb-l4n4p_9e72b48d-0d33-4e96-b726-2bc9ec75a7fc/manager/0.log"
Nov 03 21:45:25.271199 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:25.271167    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bbfc75f54-5ptcg_9fee4dbd-a32e-40e3-9e6a-20296a51776a/maas-api/0.log"
Nov 03 21:45:27.365886 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:45:27.365852    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:45:28.027511 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:28.027484    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-784fb86d65-6kg8v_1a2f424c-7201-4beb-acc4-4481306fb5a3/authorino/0.log"
Nov 03 21:45:31.478262 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:31.478233    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ccb65679-p4k6x_699e447f-e74f-4d47-ade7-df34275ab26a/kube-auth-proxy/0.log"
Nov 03 21:45:33.012120 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:33.012090    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4_cc94016b-79d8-4dd3-a968-75b7ad182092/istio-proxy/0.log"
Nov 03 21:45:33.519737 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:33.519710    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf_39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6/istio-proxy/0.log"
Nov 03 21:45:33.740066 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:33.740023    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-sfss5_be93de48-76dc-499d-905d-7d7585cde6ae/istio-proxy/0.log"
Nov 03 21:45:33.967115 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:33.967028    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-xclk7_dcc1dc11-162e-45ca-9cce-2984487898d8/istio-proxy/0.log"
Nov 03 21:45:34.425703 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:34.425670    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6b6ls_9b209916-fe0e-4b03-a767-e4f6e2e1a6c1/payload-pre-processing/0.log"
Nov 03 21:45:35.785535 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:35.785506    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw_64cb18ce-10a1-465c-9eec-ebf0cd65ffc2/main/0.log"
Nov 03 21:45:36.318509 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:36.318460    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7h9hq_739f1d75-4415-4dd2-bfcc-c3c4f83e653b/main/0.log"
Nov 03 21:45:36.853455 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:36.853425    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w_8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30/main/0.log"
Nov 03 21:45:37.135830 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:37.135755    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kksgw_be0438e3-7fe9-4c9f-a66e-4a89b436ffeb/main/0.log"
Nov 03 21:45:37.639376 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:37.639347    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p_2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac/main/0.log"
Nov 03 21:45:38.570437 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:38.570404    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-784fb86d65-6kg8v_1a2f424c-7201-4beb-acc4-4481306fb5a3/authorino/0.log"
Nov 03 21:45:47.516735 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:47.516698    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-658cf6f9cb-l4n4p_9e72b48d-0d33-4e96-b726-2bc9ec75a7fc/manager/0.log"
Nov 03 21:45:50.115021 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:50.114976    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bbfc75f54-5ptcg_9fee4dbd-a32e-40e3-9e6a-20296a51776a/maas-api/0.log"
Nov 03 21:45:52.382318 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:52.382285    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-784fb86d65-6kg8v_1a2f424c-7201-4beb-acc4-4481306fb5a3/authorino/0.log"
Nov 03 21:45:55.497238 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:45:55.497205    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ccb65679-p4k6x_699e447f-e74f-4d47-ade7-df34275ab26a/kube-auth-proxy/0.log"
Nov 03 21:45:57.365176 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:45:57.365141    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:46:07.114457 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:07.114424    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-m9vz4_cc94016b-79d8-4dd3-a968-75b7ad182092/istio-proxy/0.log"
Nov 03 21:46:07.625964 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:07.625930    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mw8gf_39ffac5b-5fae-4192-8bc3-e6fa7e84d1c6/istio-proxy/0.log"
Nov 03 21:46:07.850144 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:07.850113    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-sfss5_be93de48-76dc-499d-905d-7d7585cde6ae/istio-proxy/0.log"
Nov 03 21:46:08.091818 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:08.091795    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-xclk7_dcc1dc11-162e-45ca-9cce-2984487898d8/istio-proxy/0.log"
Nov 03 21:46:08.582030 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:08.582001    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6b6ls_9b209916-fe0e-4b03-a767-e4f6e2e1a6c1/payload-pre-processing/0.log"
Nov 03 21:46:09.652133 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:09.652101    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-d5wnw_64cb18ce-10a1-465c-9eec-ebf0cd65ffc2/main/0.log"
Nov 03 21:46:10.087202 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:10.087176    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7h9hq_739f1d75-4415-4dd2-bfcc-c3c4f83e653b/main/0.log"
Nov 03 21:46:10.572881 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:10.572853    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85g26w_8d9caaa0-3216-4a6f-90a6-a51cb7b8cd30/main/0.log"
Nov 03 21:46:10.850469 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:10.850396    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kksgw_be0438e3-7fe9-4c9f-a66e-4a89b436ffeb/main/0.log"
Nov 03 21:46:11.344989 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:46:11.344954    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fqk5p_2d4a77bd-c3f6-4bad-9b9b-3f6e4e89f8ac/main/0.log"
Nov 03 21:46:27.365624 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:46:27.365587    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:46:57.365255 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:46:57.365221    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:47:27.365665 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:47:27.365623    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:47:57.366069 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:47:57.366008    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:48:27.366438 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:48:27.366395    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:48:29.071750 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.071717    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtnjt/must-gather-mxtng"]
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072211    4274 cpu_manager.go:414] "containerMap: removing container" podUID="c631e1e6-9fb0-4e2e-b095-67584b492102" containerName="payload-pre-processing"
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072227    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c631e1e6-9fb0-4e2e-b095-67584b492102" containerName="payload-pre-processing"
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072239    4274 cpu_manager.go:414] "containerMap: removing container" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072246    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072258    4274 cpu_manager.go:414] "containerMap: removing container" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.072273 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072266    4274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.072492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072337    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c631e1e6-9fb0-4e2e-b095-67584b492102" containerName="payload-pre-processing"
Nov 03 21:48:29.072492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072345    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.072492 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.072350    4274 memory_manager.go:365] "RemoveStaleState removing state" podUID="904b5e58-d708-4645-8740-5682267e8f87" containerName="istio-proxy"
Nov 03 21:48:29.075546 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.075526    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.099088 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.099058    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqqw\" (UniqueName: \"kubernetes.io/projected/63315421-8ed5-470a-a951-bf025db3b11c-kube-api-access-9rqqw\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.099208 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.099106    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63315421-8ed5-470a-a951-bf025db3b11c-must-gather-output\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.124449 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.124423    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"openshift-service-ca.crt\""
Nov 03 21:48:29.124539 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.124497    4274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtnjt\"/\"default-dockercfg-bkvpt\""
Nov 03 21:48:29.170772 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.170744    4274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"kube-root-ca.crt\""
Nov 03 21:48:29.172433 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.172412    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/must-gather-mxtng"]
Nov 03 21:48:29.200751 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.200725    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqqw\" (UniqueName: \"kubernetes.io/projected/63315421-8ed5-470a-a951-bf025db3b11c-kube-api-access-9rqqw\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.200855 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.200789    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63315421-8ed5-470a-a951-bf025db3b11c-must-gather-output\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.201114 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.201096    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63315421-8ed5-470a-a951-bf025db3b11c-must-gather-output\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.258637 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.258587    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqqw\" (UniqueName: \"kubernetes.io/projected/63315421-8ed5-470a-a951-bf025db3b11c-kube-api-access-9rqqw\") pod \"must-gather-mxtng\" (UID: \"63315421-8ed5-470a-a951-bf025db3b11c\") " pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.388717 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.388645    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/must-gather-mxtng"
Nov 03 21:48:29.754408 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.754381    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/must-gather-mxtng"]
Nov 03 21:48:29.761603 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.761576    4274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:48:29.879317 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:29.879278    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/must-gather-mxtng" event={"ID":"63315421-8ed5-470a-a951-bf025db3b11c","Type":"ContainerStarted","Data":"b9da6c07051eeaa3c6b5119c0f6017a53738bf585100eae598f07b59aea933e9"}
Nov 03 21:48:31.889435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:31.889401    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/must-gather-mxtng" event={"ID":"63315421-8ed5-470a-a951-bf025db3b11c","Type":"ContainerStarted","Data":"908444d21b1cb94b84cb461e13fdf5a8becfc3d49c35b3271e458fc3d1743368"}
Nov 03 21:48:31.889435 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:31.889441    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/must-gather-mxtng" event={"ID":"63315421-8ed5-470a-a951-bf025db3b11c","Type":"ContainerStarted","Data":"faf6672b8ef647566c445883df83c43398732e66e49bcc7782d2ea349a7477b1"}
Nov 03 21:48:31.998433 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:31.998379    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mtnjt/must-gather-mxtng" podStartSLOduration=1.587477499 podStartE2EDuration="2.99836769s" totalImagesPullingTime="1.410890191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:31.996788332 +0000 UTC m=+4736.923358565" watchObservedRunningTime="2026-11-03 21:48:31.99836769 +0000 UTC m=+4736.924937927"
Nov 03 21:48:38.867310 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:38.867271    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m7s5v_038206a5-8ba4-456e-8e71-db0ec9cdd46c/global-pull-secret-syncer/0.log"
Nov 03 21:48:42.459246 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:42.459211    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rqnkl_c243550e-f884-41f5-986f-69456856c89f/konnectivity-agent/0.log"
Nov 03 21:48:42.984901 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:48:42.984875    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-118.us-west-2.compute.internal_1c4b446c071455ed0e37e47646678b6e/haproxy/0.log"
Nov 03 21:48:57.365886 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:48:57.365836    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:49:07.517337 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.517294    4274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"]
Nov 03 21:49:07.527067 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.527024    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.531595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.531404    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-sys\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.531595 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.531527    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlz7\" (UniqueName: \"kubernetes.io/projected/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-kube-api-access-gzlz7\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.531790 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.531596    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-podres\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.531790 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.531699    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-lib-modules\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.531935 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.531787    4274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-proc\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.574079 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.574053    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-784fb86d65-6kg8v_1a2f424c-7201-4beb-acc4-4481306fb5a3/authorino/0.log"
Nov 03 21:49:07.607061 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.607004    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"]
Nov 03 21:49:07.632467 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632437    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-proc\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632494    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-sys\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632525    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlz7\" (UniqueName: \"kubernetes.io/projected/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-kube-api-access-gzlz7\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632541    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-podres\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632577    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-proc\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632623 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632601    4274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-lib-modules\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632655    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-sys\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632713    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-podres\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.632843 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.632755    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-lib-modules\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.689443 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.689417    4274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlz7\" (UniqueName: \"kubernetes.io/projected/998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7-kube-api-access-gzlz7\") pod \"perf-node-gather-daemonset-ltnhm\" (UID: \"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:07.851221 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:07.851149    4274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:08.214060 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:08.213975    4274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"]
Nov 03 21:49:08.216355 ip-10-0-128-118 kubenswrapper[4274]: W1103 21:49:08.216319    4274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod998c8cb4_7ac2_47de_9fa9_cab64d5cb8d7.slice/crio-189aa6b793c8e05e7c6ed4270c0c52d204a27e404ba7c4847d58c3c01858b927 WatchSource:0}: Error finding container 189aa6b793c8e05e7c6ed4270c0c52d204a27e404ba7c4847d58c3c01858b927: Status 404 returned error can't find the container with id 189aa6b793c8e05e7c6ed4270c0c52d204a27e404ba7c4847d58c3c01858b927
Nov 03 21:49:09.101493 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:09.101449    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm" event={"ID":"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7","Type":"ContainerStarted","Data":"03b69dad9f036a34f9f446dc4047348ebc486d79970c79f2a4598bcb9f72d260"}
Nov 03 21:49:09.101946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:09.101521    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:09.101946 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:09.101547    4274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm" event={"ID":"998c8cb4-7ac2-47de-9fa9-cab64d5cb8d7","Type":"ContainerStarted","Data":"189aa6b793c8e05e7c6ed4270c0c52d204a27e404ba7c4847d58c3c01858b927"}
Nov 03 21:49:09.210381 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:09.210332    4274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm" podStartSLOduration=2.210314375 podStartE2EDuration="2.210314375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:09.208558073 +0000 UTC m=+4774.135128312" watchObservedRunningTime="2026-11-03 21:49:09.210314375 +0000 UTC m=+4774.136884613"
Nov 03 21:49:15.116996 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:15.116960    4274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-ltnhm"
Nov 03 21:49:16.494291 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:16.494254    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bqp2m_8db69f86-491b-451b-b4d1-957b106365dc/kube-state-metrics/0.log"
Nov 03 21:49:16.897122 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:16.897092    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bqp2m_8db69f86-491b-451b-b4d1-957b106365dc/kube-rbac-proxy-main/0.log"
Nov 03 21:49:17.302805 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:17.302776    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bqp2m_8db69f86-491b-451b-b4d1-957b106365dc/kube-rbac-proxy-self/0.log"
Nov 03 21:49:18.558768 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:18.558742    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qfl7_163948e9-a5fc-4339-bee1-0a409da473c8/node-exporter/0.log"
Nov 03 21:49:18.964227 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:18.964160    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qfl7_163948e9-a5fc-4339-bee1-0a409da473c8/kube-rbac-proxy/0.log"
Nov 03 21:49:19.369978 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:19.369950    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qfl7_163948e9-a5fc-4339-bee1-0a409da473c8/init-textfile/0.log"
Nov 03 21:49:27.366377 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:49:27.366342    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"
Nov 03 21:49:27.470866 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:27.470839    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-89pt2_deb2e161-ec98-4f5d-9da4-c648984b9ae8/prometheus-operator-admission-webhook/0.log"
Nov 03 21:49:27.880919 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:27.880888    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b9685c66-sdgxz_2feccb21-77ca-40e0-975b-2c0492f4286f/telemeter-client/0.log"
Nov 03 21:49:28.286114 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:28.286084    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b9685c66-sdgxz_2feccb21-77ca-40e0-975b-2c0492f4286f/reload/0.log"
Nov 03 21:49:28.694125 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:28.694013    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b9685c66-sdgxz_2feccb21-77ca-40e0-975b-2c0492f4286f/kube-rbac-proxy/0.log"
Nov 03 21:49:39.151326 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:39.151301    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76749cb79-md28w_0093d903-174d-4539-98bf-55ad96afac22/console/0.log"
Nov 03 21:49:39.571531 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:39.571499    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-n6k5m_3d788f8d-574b-4604-b1b2-045ce07022f1/download-server/0.log"
Nov 03 21:49:48.389204 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:48.389129    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pmkxf_2664af0a-7726-48b6-83df-96197730d5dc/dns/0.log"
Nov 03 21:49:48.791637 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:48.791597    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pmkxf_2664af0a-7726-48b6-83df-96197730d5dc/kube-rbac-proxy/0.log"
Nov 03 21:49:50.014480 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:50.014455    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c6pqw_a1aff89a-851d-4a9a-9b60-63c75683b36a/dns-node-resolver/0.log"
Nov 03 21:49:53.550383 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:53.550353    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-665c494c89-mpwvq_5f595b4e-61b2-442e-b8e8-3790ada9214a/registry/0.log"
Nov 03 21:49:54.894181 ip-10-0-128-118 kubenswrapper[4274]: I1103 21:49:54.894147    4274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-njl4k_fa85e6e6-4c02-4730-ac6d-ab7e8b6f82da/node-ca/0.log"
Nov 03 21:49:57.364773 ip-10-0-128-118 kubenswrapper[4274]: E1103 21:49:57.364740    4274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w88bv" containerName="extractor"